2024-03-26T05:04:38.637 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T05:04:38.637 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075 --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075/orig.config.yaml 2024-03-26T05:04:38.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T05:04:38.738 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075 branch: squid description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7622075' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a extra_packages: - cephadm selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFRp4++OagDBvNa6ehQ4JmsCV/v+B5kUytsBomIUZl00QzXyrQFBndK9W5yc2eGIVqbvucpVVQqy1dNt5DuKTs0= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJIVMUQQX43jTvJYdvLMDQdSAdHN+8pNCO3H9rul/FAsqINti7ogSOQy5l2C+nGSgFTGSx8RlU/5Skodg5aoPFE= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T05:04:38.738 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T05:04:38.739 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T05:04:38.739 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T05:04:38.740 INFO:teuthology.task.internal:Checking packages... 2024-03-26T05:04:38.765 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T05:04:38.765 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T05:04:38.765 INFO:teuthology.packaging:ref: None 2024-03-26T05:04:38.765 INFO:teuthology.packaging:tag: None 2024-03-26T05:04:38.765 INFO:teuthology.packaging:branch: squid 2024-03-26T05:04:38.765 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T05:04:38.766 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T05:04:38.923 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T05:04:38.924 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T05:04:38.932 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T05:04:38.932 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T05:04:38.939 INFO:teuthology.task.internal:Saving configuration 2024-03-26T05:04:38.950 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T05:04:38.957 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T05:04:38.981 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:59:21.852740', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFRp4++OagDBvNa6ehQ4JmsCV/v+B5kUytsBomIUZl00QzXyrQFBndK9W5yc2eGIVqbvucpVVQqy1dNt5DuKTs0='} 2024-03-26T05:04:39.006 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:59:21.854509', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJIVMUQQX43jTvJYdvLMDQdSAdHN+8pNCO3H9rul/FAsqINti7ogSOQy5l2C+nGSgFTGSx8RlU/5Skodg5aoPFE='} 2024-03-26T05:04:39.006 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T05:04:39.013 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T05:04:39.013 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T05:04:39.014 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T05:04:39.093 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdd8db50f70>, signals=[15]) 2024-03-26T05:04:39.093 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T05:04:39.099 INFO:teuthology.task.internal:Opening connections... 2024-03-26T05:04:39.099 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-03-26T05:04:39.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:04:39.194 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-03-26T05:04:39.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:04:39.284 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T05:04:39.291 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-03-26T05:04:39.308 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-03-26T05:04:39.309 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-03-26T05:04:39.365 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-03-26T05:04:39.366 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-03-26T05:04:39.366 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-03-26T05:04:39.366 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-03-26T05:04:39.366 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-03-26T05:04:39.366 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-03-26T05:04:39.366 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T05:04:39.366 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-03-26T05:04:39.367 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-03-26T05:04:39.367 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T05:04:39.367 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-03-26T05:04:39.367 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T05:04:39.367 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T05:04:39.367 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T05:04:39.367 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-03-26T05:04:39.394 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-03-26T05:04:39.413 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-03-26T05:04:39.413 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-03-26T05:04:39.469 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-03-26T05:04:39.469 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-03-26T05:04:39.469 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-03-26T05:04:39.469 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-03-26T05:04:39.469 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-03-26T05:04:39.469 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-03-26T05:04:39.469 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T05:04:39.470 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-03-26T05:04:39.470 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-03-26T05:04:39.470 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T05:04:39.470 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-03-26T05:04:39.470 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T05:04:39.470 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T05:04:39.470 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T05:04:39.470 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-03-26T05:04:39.492 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T05:04:39.502 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T05:04:39.508 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T05:04:39.508 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-03-26T05:04:39.511 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-03-26T05:04:39.526 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T05:04:39.532 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T05:04:39.532 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-03-26T05:04:39.568 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-03-26T05:04:39.620 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T05:04:39.673 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T05:04:39.687 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T05:04:39.687 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T05:04:39.688 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T05:04:39.688 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T05:04:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.704 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-03-26T05:04:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:04:39.705 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-26T05:04:39.760 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-361.el9.x86_64 2024-03-26T05:04:39.760 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.14.0-361.el9.x86_64 2024-03-26T05:04:39.760 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2024-03-26T05:04:39.761 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-361.el9.x86_64 2024-03-26T05:04:39.762 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-361.el9.x86_64 2024-03-26T05:04:39.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-03-26T05:04:41.863 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-26T05:04:42.173 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 8.5 MB/s | 6.7 MB 00:00 2024-03-26T05:04:43.464 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-26T05:04:43.798 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-26T05:04:45.564 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-03-26T05:04:47.581 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-26T05:04:47.773 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-03-26T05:04:49.613 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-26T05:04:51.457 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-26T05:04:53.816 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-26T05:04:56.120 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 21 MB 00:02 2024-03-26T05:04:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-26T05:04:58.222 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-03-26T05:05:01.226 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 7.7 kB/s | 1.7 kB 00:00 2024-03-26T05:05:03.416 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T05:05:03.416 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T05:05:03.512 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T05:05:03.517 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:05:03.518 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:05:03.518 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:05:03.518 INFO:teuthology.orchestra.run.smithi164.stdout:Install 4 Packages 2024-03-26T05:05:03.518 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:05:03.518 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 76 M 2024-03-26T05:05:03.518 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 123 M 2024-03-26T05:05:03.519 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:05:03.627 INFO:teuthology.orchestra.run.smithi164.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 910 kB/s | 66 kB 00:00 2024-03-26T05:05:04.402 INFO:teuthology.orchestra.run.smithi164.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-26T05:05:04.786 INFO:teuthology.orchestra.run.smithi164.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-26T05:05:06.444 INFO:teuthology.orchestra.run.smithi164.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 9.8 MB/s | 27 MB 00:02 2024-03-26T05:05:06.445 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:05:06.445 INFO:teuthology.orchestra.run.smithi164.stdout:Total 26 MB/s | 76 MB 00:02 2024-03-26T05:05:07.115 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:05:07.471 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:05:07.471 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:05:08.467 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:05:08.468 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:05:10.925 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:05:11.524 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T05:05:11.627 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T05:05:14.367 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T05:05:14.505 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T05:05:18.823 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-26T05:05:19.188 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T05:05:19.305 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:05:23.795 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T05:05:24.023 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:05:25.968 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T05:05:25.968 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T05:05:26.171 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:05:26.176 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:05:26.176 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 76 M 2024-03-26T05:05:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 123 M 2024-03-26T05:05:26.178 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:05:26.394 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:05:26.395 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:05:26.395 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:05:26.561 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:05:26.580 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:05:26.597 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:05:26.963 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 101 kB/s | 66 kB 00:00 2024-03-26T05:05:27.006 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:05:27.040 INFO:teuthology.orchestra.run.smithi136.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:05:32.129 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 2.6 MB/s | 15 MB 00:05 2024-03-26T05:05:38.703 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 2.7 MB/s | 33 MB 00:12 2024-03-26T05:05:48.209 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 1.3 MB/s | 27 MB 00:21 2024-03-26T05:05:48.213 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:05:48.213 INFO:teuthology.orchestra.run.smithi136.stdout:Total 3.4 MB/s | 76 MB 00:22 2024-03-26T05:05:48.877 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:05:49.286 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:05:49.287 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:05:50.281 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:05:50.282 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:05:52.729 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:05:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T05:05:53.355 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:05:53.355 INFO:teuthology.orchestra.run.smithi164.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T05:05:53.355 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:05:53.400 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:05:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T05:05:53.738 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:05:53.738 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T05:05:53.738 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T05:05:53.738 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T05:05:54.154 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:05:54.155 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:05:54.155 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:05:54.155 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T05:05:54.155 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T05:05:54.155 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T05:05:54.155 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T05:05:54.155 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:05:54.155 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:05:54.500 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T05:05:54.613 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T05:05:54.614 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T05:05:54.614 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T05:05:54.614 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T05:05:54.614 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T05:05:54.614 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T05:05:54.614 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T05:05:54.614 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T05:05:54.694 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T05:05:54.694 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T05:05:54.695 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2024-03-26T05:05:54.738 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2024-03-26T05:05:54.738 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.06 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 68.el9 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Group : Unspecified 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8264254 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : builder@centos.org 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2024-03-26T05:05:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:05:54.740 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T05:05:54.741 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T05:05:54.741 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 5.14.0-432.el9.x86_64 2024-03-26T05:05:54.741 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T05:05:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-03-26T05:05:56.214 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T05:05:56.349 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T05:05:56.756 INFO:teuthology.orchestra.run.smithi164.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T05:05:56.793 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-03-26T05:05:56.796 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-03-26T05:05:56.812 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.od9XJKGicj 2024-03-26T05:05:56.812 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.od9XJKGicj 2024-03-26T05:05:56.882 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.od9XJKGicj 2024-03-26T05:05:57.020 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.od9XJKGicj is 6KB 2024-03-26T05:05:57.068 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.od9XJKGicj 2024-03-26T05:05:57.082 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T05:05:57.149 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T05:05:57.149 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T05:05:57.149 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T05:05:57.149 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T05:05:57.150 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T05:05:57.294 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-03-26T05:06:01.091 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T05:06:01.125 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:06:05.899 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:06:27.298 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T05:06:27.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T05:06:27.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:06:35.150 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:06:35.151 INFO:teuthology.orchestra.run.smithi136.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T05:06:35.151 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:06:35.194 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:06:35.576 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:06:35.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T05:06:35.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T05:06:35.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:06:36.012 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:06:36.373 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T05:06:36.470 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T05:06:36.470 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T05:06:36.470 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T05:06:36.471 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T05:06:36.471 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T05:06:36.471 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T05:06:36.471 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T05:06:36.471 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T05:06:36.551 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T05:06:36.551 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T05:06:36.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-03-26T05:06:36.585 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-03-26T05:06:36.586 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-03-26T05:06:36.586 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-03-26T05:06:36.586 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 68.el9 2024-03-26T05:06:36.586 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-03-26T05:06:36.586 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T05:06:36.587 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-03-26T05:06:36.587 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8264254 2024-03-26T05:06:36.587 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-03-26T05:06:36.587 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T05:06:36.587 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T05:06:36.587 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T05:06:36.587 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T05:06:36.588 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-03-26T05:06:36.588 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-03-26T05:06:36.588 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T05:06:36.588 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-03-26T05:06:36.588 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-03-26T05:06:36.588 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:06:36.589 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T05:06:36.589 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T05:06:36.589 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T05:06:36.590 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-03-26T05:06:36.590 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:06:36.590 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T05:06:36.591 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T05:06:36.592 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-432.el9.x86_64 2024-03-26T05:06:36.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T05:06:37.180 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-03-26T05:06:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T05:06:38.698 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-03-26T05:06:38.700 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-26T05:06:38.715 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.ydXO8MexVB 2024-03-26T05:06:38.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ydXO8MexVB 2024-03-26T05:06:38.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.ydXO8MexVB 2024-03-26T05:06:38.931 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.ydXO8MexVB is 6KB 2024-03-26T05:06:38.981 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.ydXO8MexVB 2024-03-26T05:06:38.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T05:06:39.059 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T05:06:39.059 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T05:06:39.059 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T05:06:39.059 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T05:06:39.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T05:06:39.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-03-26T05:07:01.791 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-26T05:07:09.224 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T05:07:09.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T05:07:09.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:07:10.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T05:07:10.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:07:29.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-26T05:07:41.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T05:07:41.196 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:07:44.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-26T05:07:59.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T05:07:59.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:07:59.552 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-26T05:07:59.947 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T05:07:59.947 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T05:07:59.948 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-03-26T05:07:59.963 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-432.el9.x86_64 2024-03-26T05:07:59.963 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T05:07:59.963 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T05:07:59.963 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T05:08:00.964 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T05:08:00.964 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T05:08:01.057 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-03-26T05:08:01.088 DEBUG:teuthology.parallel:result is None 2024-03-26T05:08:09.227 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T05:08:18.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T05:08:18.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:08:18.672 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-26T05:08:19.238 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T05:08:19.238 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T05:08:19.239 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-26T05:08:19.253 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-432.el9.x86_64 2024-03-26T05:08:19.253 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T05:08:19.253 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T05:08:19.253 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T05:08:20.254 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T05:08:20.254 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T05:08:20.349 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-03-26T05:08:20.376 DEBUG:teuthology.parallel:result is None 2024-03-26T05:08:20.377 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T05:08:20.385 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T05:08:20.385 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T05:08:20.388 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T05:08:20.406 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T05:08:20.412 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T05:08:20.418 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T05:08:20.418 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T05:08:20.442 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T05:08:20.476 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T05:08:20.482 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T05:08:20.483 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T05:08:20.498 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T05:08:20.548 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T05:08:20.549 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T05:08:20.562 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T05:08:20.563 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T05:08:20.564 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T05:08:20.571 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T05:08:20.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T05:08:20.573 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T05:08:20.634 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T05:08:20.643 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T05:08:20.644 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T05:08:20.646 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T05:08:20.690 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T05:08:20.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T05:08:20.788 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T05:08:20.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T05:08:20.911 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:08:20.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T05:08:20.978 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T05:08:21.023 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T05:08:21.079 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T05:08:21.164 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T05:08:21.206 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:08:21.206 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T05:08:21.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-03-26T05:08:21.277 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-03-26T05:08:21.327 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T05:08:21.356 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T05:08:21.756 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T05:08:21.764 INFO:teuthology.task.internal:Starting timer... 2024-03-26T05:08:21.764 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T05:08:21.784 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T05:08:21.793 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T05:08:21.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-03-26T05:08:21.845 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-03-26T05:08:21.847 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-03-26T05:08:21.903 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-03-26T05:08:21.905 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T05:08:21.905 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-03-26T05:08:21.920 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-03-26T05:08:21.920 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-03-26T05:08:21.947 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-03-26T05:08:21.947 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi136.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2024-03-26T05:08:21.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T05:08:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:21.989 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T05:08:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:08:22.017 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T05:08:22.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-03-26T05:08:22.072 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-03-26T05:08:22.116 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T05:08:22.131 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-26T05:08:22.138 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-26T05:08:22.267 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T05:08:22.283 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T05:08:22.284 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi136.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T05:12:48.423 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi136.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-03-26T05:12:48.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T05:12:48.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:12:48.517 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-26T05:12:48.593 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-26T05:12:48.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T05:12:48.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T05:12:48.682 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-26T05:12:48.764 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T05:12:48.765 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T05:12:48.775 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T05:12:48.775 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T05:12:48.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T05:12:48.781 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T05:12:48.781 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T05:12:48.809 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T05:12:48.826 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T05:12:48.834 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T05:12:48.851 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T05:12:48.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-03-26T05:12:48.871 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-03-26T05:12:48.880 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-03-26T05:12:48.887 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T05:12:48.893 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-03-26T05:12:48.904 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T05:12:48.910 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T05:12:48.926 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T05:12:48.964 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-26T05:12:48.968 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T05:12:48.968 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-26T05:12:48.968 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:12:48.968 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:12:48.968 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:12:48.968 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:12:48.977 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-26T05:12:48.981 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T05:12:48.981 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-26T05:12:48.981 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:12:48.981 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:12:48.981 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:12:48.981 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:12:48.981 INFO:teuthology.run_tasks:Running task install... 2024-03-26T05:12:48.990 DEBUG:teuthology.task.install:project ceph 2024-03-26T05:12:48.990 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'}, 'extra_packages': ['cephadm']} 2024-03-26T05:12:48.990 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T05:12:48.990 INFO:teuthology.task.install:Using flavor: default 2024-03-26T05:12:48.996 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T05:12:48.996 INFO:teuthology.task.install:extra packages: [] 2024-03-26T05:12:48.996 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T05:12:48.997 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T05:12:48.999 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T05:12:48.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T05:12:49.150 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T05:12:49.150 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T05:12:49.259 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T05:12:49.259 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:12:49.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T05:12:49.287 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T05:12:49.287 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T05:12:49.307 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T05:12:49.307 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T05:12:49.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T05:12:49.438 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T05:12:49.481 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T05:12:49.481 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:12:49.481 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T05:12:49.503 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-03-26T05:12:49.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-03-26T05:12:49.510 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T05:12:49.510 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T05:12:49.579 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T05:12:49.659 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T05:12:49.726 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2024-03-26T05:12:49.728 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2024-03-26T05:12:49.753 INFO:teuthology.orchestra.run.smithi136.stdout:74 files removed 2024-03-26T05:12:49.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw 2024-03-26T05:12:49.969 INFO:teuthology.orchestra.run.smithi164.stdout:74 files removed 2024-03-26T05:12:49.996 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw 2024-03-26T05:12:50.494 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 201 kB/s | 83 kB 00:00 2024-03-26T05:12:50.698 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-03-26T05:12:50.861 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-26T05:12:51.081 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-26T05:12:51.212 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-26T05:12:51.474 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-03-26T05:12:51.756 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T05:12:51.876 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-26T05:12:53.122 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-26T05:12:53.526 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-26T05:12:55.252 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-03-26T05:12:57.169 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-26T05:12:57.621 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-26T05:13:00.147 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-26T05:13:02.219 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-26T05:13:03.653 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-26T05:13:06.074 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-26T05:13:07.498 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-26T05:13:15.193 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 2.7 MB/s | 21 MB 00:07 2024-03-26T05:13:20.264 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 6.6 kB/s | 1.7 kB 00:00 2024-03-26T05:13:22.428 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-03-26T05:13:22.732 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:13:22.745 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:13:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:13:22.745 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:13:22.745 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:13:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T05:13:22.745 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-03-26T05:13:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T05:13:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T05:13:22.746 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:13:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options 2024-03-26T05:13:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T05:13:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T05:13:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T05:13:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T05:13:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-03-26T05:13:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T05:13:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:22.749 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:13:22.749 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:13:22.749 INFO:teuthology.orchestra.run.smithi136.stdout:Install 30 Packages 2024-03-26T05:13:22.749 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-03-26T05:13:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:22.749 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 58 M 2024-03-26T05:13:22.750 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:13:24.432 INFO:teuthology.orchestra.run.smithi136.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 3.9 MB/s | 5.3 MB 00:01 2024-03-26T05:13:24.515 INFO:teuthology.orchestra.run.smithi136.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 302 kB/s | 25 kB 00:00 2024-03-26T05:13:24.682 INFO:teuthology.orchestra.run.smithi136.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 4.4 MB/s | 744 kB 00:00 2024-03-26T05:13:24.816 INFO:teuthology.orchestra.run.smithi136.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 3.7 MB/s | 498 kB 00:00 2024-03-26T05:13:25.000 INFO:teuthology.orchestra.run.smithi136.stdout:(5/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9. 5.1 MB/s | 9.8 MB 00:01 2024-03-26T05:13:25.091 INFO:teuthology.orchestra.run.smithi136.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 494 kB/s | 45 kB 00:00 2024-03-26T05:13:25.225 INFO:teuthology.orchestra.run.smithi136.stdout:(7/32): python3-ceph-common-19.0.0-1775.g5e7012 984 kB/s | 131 kB 00:00 2024-03-26T05:13:25.309 INFO:teuthology.orchestra.run.smithi136.stdout:(8/32): python3-cephfs-19.0.0-1775.g5e70124b.el 1.9 MB/s | 162 kB 00:00 2024-03-26T05:13:25.425 INFO:teuthology.orchestra.run.smithi136.stdout:(9/32): python3-rados-19.0.0-1775.g5e70124b.el9 2.7 MB/s | 321 kB 00:00 2024-03-26T05:13:25.509 INFO:teuthology.orchestra.run.smithi136.stdout:(10/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_ 7.4 MB/s | 5.1 MB 00:00 2024-03-26T05:13:25.534 INFO:teuthology.orchestra.run.smithi136.stdout:(11/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 2.7 MB/s | 298 kB 00:00 2024-03-26T05:13:25.593 INFO:teuthology.orchestra.run.smithi136.stdout:(12/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.2 MB/s | 99 kB 00:00 2024-03-26T05:13:25.793 INFO:teuthology.orchestra.run.smithi136.stdout:(13/32): ceph-common-19.0.0-1775.g5e70124b.el9. 7.5 MB/s | 20 MB 00:02 2024-03-26T05:13:25.835 INFO:teuthology.orchestra.run.smithi136.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 352 kB/s | 106 kB 00:00 2024-03-26T05:13:25.877 INFO:teuthology.orchestra.run.smithi136.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 563 kB/s | 159 kB 00:00 2024-03-26T05:13:25.894 INFO:teuthology.orchestra.run.smithi136.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 440 kB/s | 44 kB 00:00 2024-03-26T05:13:25.970 INFO:teuthology.orchestra.run.smithi136.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-26T05:13:26.011 INFO:teuthology.orchestra.run.smithi136.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-26T05:13:26.028 INFO:teuthology.orchestra.run.smithi136.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 308 kB/s | 41 kB 00:00 2024-03-26T05:13:26.137 INFO:teuthology.orchestra.run.smithi136.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 332 kB/s | 36 kB 00:00 2024-03-26T05:13:26.153 INFO:teuthology.orchestra.run.smithi136.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 380 kB/s | 47 kB 00:00 2024-03-26T05:13:26.270 INFO:teuthology.orchestra.run.smithi136.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 736 kB/s | 221 kB 00:00 2024-03-26T05:13:26.312 INFO:teuthology.orchestra.run.smithi136.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-03-26T05:13:26.329 INFO:teuthology.orchestra.run.smithi136.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-03-26T05:13:26.346 INFO:teuthology.orchestra.run.smithi136.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-26T05:13:26.396 INFO:teuthology.orchestra.run.smithi136.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-26T05:13:26.472 INFO:teuthology.orchestra.run.smithi136.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-26T05:13:26.522 INFO:teuthology.orchestra.run.smithi136.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-26T05:13:26.722 INFO:teuthology.orchestra.run.smithi136.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-03-26T05:13:26.923 INFO:teuthology.orchestra.run.smithi136.stdout:(30/32): librados2-19.0.0-1775.g5e70124b.el9.x8 8.5 MB/s | 3.4 MB 00:00 2024-03-26T05:13:27.007 INFO:teuthology.orchestra.run.smithi136.stdout:(31/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-26T05:13:27.394 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-03-26T05:13:27.473 INFO:teuthology.orchestra.run.smithi136.stdout:(32/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 4.2 MB/s | 3.1 MB 00:00 2024-03-26T05:13:27.475 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:13:27.476 INFO:teuthology.orchestra.run.smithi136.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-26T05:13:27.511 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:13:27.549 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:13:27.549 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:13:28.035 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:13:28.036 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:13:29.560 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:13:29.653 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T05:13:29.686 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T05:13:29.711 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T05:13:29.743 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T05:13:29.759 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T05:13:29.864 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T05:13:29.877 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T05:13:29.878 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T05:13:29.879 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout:Install 30 Packages 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 2 Packages 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:29.880 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 58 M 2024-03-26T05:13:29.881 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:13:29.943 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T05:13:29.999 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T05:13:30.341 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T05:13:30.396 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T05:13:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T05:13:30.569 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T05:13:30.598 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T05:13:30.660 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T05:13:30.697 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T05:13:30.746 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T05:13:30.819 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T05:13:30.841 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T05:13:30.887 INFO:teuthology.orchestra.run.smithi164.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 7.4 MB/s | 5.3 MB 00:00 2024-03-26T05:13:30.888 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T05:13:30.923 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T05:13:30.929 INFO:teuthology.orchestra.run.smithi164.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 605 kB/s | 25 kB 00:00 2024-03-26T05:13:31.021 INFO:teuthology.orchestra.run.smithi164.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 7.9 MB/s | 744 kB 00:00 2024-03-26T05:13:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T05:13:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T05:13:31.087 INFO:teuthology.orchestra.run.smithi164.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 7.3 MB/s | 498 kB 00:00 2024-03-26T05:13:31.106 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T05:13:31.139 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T05:13:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T05:13:31.200 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T05:13:31.504 INFO:teuthology.orchestra.run.smithi164.stdout:(5/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_6 12 MB/s | 5.1 MB 00:00 2024-03-26T05:13:31.515 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T05:13:31.546 INFO:teuthology.orchestra.run.smithi164.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 1.1 MB/s | 45 kB 00:00 2024-03-26T05:13:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T05:13:31.605 INFO:teuthology.orchestra.run.smithi164.stdout:(7/32): python3-ceph-common-19.0.0-1775.g5e7012 2.2 MB/s | 131 kB 00:00 2024-03-26T05:13:31.637 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T05:13:31.664 INFO:teuthology.orchestra.run.smithi164.stdout:(8/32): python3-cephfs-19.0.0-1775.g5e70124b.el 2.7 MB/s | 162 kB 00:00 2024-03-26T05:13:31.668 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T05:13:31.711 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T05:13:31.739 INFO:teuthology.orchestra.run.smithi164.stdout:(9/32): python3-rados-19.0.0-1775.g5e70124b.el9 4.2 MB/s | 321 kB 00:00 2024-03-26T05:13:31.806 INFO:teuthology.orchestra.run.smithi164.stdout:(10/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 4.4 MB/s | 298 kB 00:00 2024-03-26T05:13:31.865 INFO:teuthology.orchestra.run.smithi164.stdout:(11/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.7 MB/s | 99 kB 00:00 2024-03-26T05:13:32.006 INFO:teuthology.orchestra.run.smithi164.stdout:(12/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9 5.3 MB/s | 9.8 MB 00:01 2024-03-26T05:13:32.065 INFO:teuthology.orchestra.run.smithi164.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 528 kB/s | 106 kB 00:00 2024-03-26T05:13:32.090 INFO:teuthology.orchestra.run.smithi164.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-26T05:13:32.107 INFO:teuthology.orchestra.run.smithi164.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-26T05:13:32.158 INFO:teuthology.orchestra.run.smithi164.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-26T05:13:32.170 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T05:13:32.183 INFO:teuthology.orchestra.run.smithi164.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-26T05:13:32.200 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T05:13:32.200 INFO:teuthology.orchestra.run.smithi164.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 994 kB/s | 41 kB 00:00 2024-03-26T05:13:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T05:13:32.279 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:13:32.280 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:13:32.330 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T05:13:32.389 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T05:13:32.409 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:13:32.625 INFO:teuthology.orchestra.run.smithi164.stdout:(19/32): ceph-common-19.0.0-1775.g5e70124b.el9. 8.3 MB/s | 20 MB 00:02 2024-03-26T05:13:32.626 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:13:32.685 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:13:32.726 INFO:teuthology.orchestra.run.smithi164.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 68 kB/s | 36 kB 00:00 2024-03-26T05:13:32.726 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T05:13:32.759 INFO:teuthology.orchestra.run.smithi164.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 383 kB/s | 221 kB 00:00 2024-03-26T05:13:32.851 INFO:teuthology.orchestra.run.smithi164.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 210 kB/s | 47 kB 00:00 2024-03-26T05:13:32.993 INFO:teuthology.orchestra.run.smithi164.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 184 kB/s | 26 kB 00:00 2024-03-26T05:13:33.018 INFO:teuthology.orchestra.run.smithi164.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-26T05:13:33.085 INFO:teuthology.orchestra.run.smithi164.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-26T05:13:33.110 INFO:teuthology.orchestra.run.smithi164.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-03-26T05:13:33.202 INFO:teuthology.orchestra.run.smithi164.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-26T05:13:33.245 INFO:teuthology.orchestra.run.smithi164.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-26T05:13:33.370 INFO:teuthology.orchestra.run.smithi164.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-26T05:13:33.437 INFO:teuthology.orchestra.run.smithi164.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.8 MB/s | 4.4 MB 00:00 2024-03-26T05:13:33.587 INFO:teuthology.orchestra.run.smithi164.stdout:(31/32): librados2-19.0.0-1775.g5e70124b.el9.x8 9.9 MB/s | 3.4 MB 00:00 2024-03-26T05:13:34.070 INFO:teuthology.orchestra.run.smithi164.stdout:(32/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 4.5 MB/s | 3.1 MB 00:00 2024-03-26T05:13:34.074 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:13:34.074 INFO:teuthology.orchestra.run.smithi164.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-26T05:13:34.106 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:13:34.141 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:13:34.141 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:13:34.573 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T05:13:34.606 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T05:13:34.631 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:13:34.631 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:13:35.156 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T05:13:35.193 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T05:13:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T05:13:35.268 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T05:13:35.268 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T05:13:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T05:13:36.090 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:13:36.181 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T05:13:36.261 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T05:13:36.301 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T05:13:36.324 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T05:13:36.341 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T05:13:36.517 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T05:13:36.571 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T05:13:36.899 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T05:13:36.961 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T05:13:37.035 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T05:13:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T05:13:37.145 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T05:13:37.216 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T05:13:37.245 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T05:13:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T05:13:37.357 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T05:13:37.381 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T05:13:37.445 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T05:13:37.480 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T05:13:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T05:13:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T05:13:37.681 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T05:13:37.712 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T05:13:37.741 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T05:13:37.790 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T05:13:38.098 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T05:13:38.129 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T05:13:38.219 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T05:13:38.259 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T05:13:38.300 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T05:13:38.751 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T05:13:38.782 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T05:13:38.852 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T05:13:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T05:13:38.950 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T05:13:41.173 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T05:13:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T05:13:41.760 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T05:13:41.800 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T05:13:41.879 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T05:13:41.879 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T05:13:41.879 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T05:13:41.879 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:41.918 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T05:13:45.098 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T05:13:45.099 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-03-26T05:13:45.099 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-03-26T05:13:45.099 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-03-26T05:13:45.099 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-03-26T05:13:45.099 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-03-26T05:13:45.099 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-03-26T05:13:45.099 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-03-26T05:13:45.099 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T05:13:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T05:13:46.142 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:13:46.142 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T05:13:46.143 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T05:13:46.143 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T05:13:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T05:13:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T05:13:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T05:13:47.353 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T05:13:47.354 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T05:13:47.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T05:13:48.530 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T05:13:48.531 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.532 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:48.533 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:13:48.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test 2024-03-26T05:13:49.446 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:29 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:13:49.603 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:13:49.603 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T05:13:49.604 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T05:13:49.605 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:49.605 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:13:49.605 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:13:49.605 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-03-26T05:13:49.605 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:13:49.605 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 48 M 2024-03-26T05:13:49.606 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 205 M 2024-03-26T05:13:49.606 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:13:49.835 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-26T05:13:49.885 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-26T05:13:49.911 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-26T05:13:50.044 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-26T05:13:50.078 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 401 kB/s | 163 kB 00:00 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2024-03-26T05:13:51.475 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:52.371 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T05:13:52.395 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T05:13:52.396 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:13:52.396 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T05:13:52.396 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T05:13:52.396 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T05:13:52.396 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:52.398 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T05:13:52.481 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T05:13:52.482 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T05:13:52.920 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 15 MB/s | 47 MB 00:03 2024-03-26T05:13:52.923 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:13:52.923 INFO:teuthology.orchestra.run.smithi136.stdout:Total 15 MB/s | 48 MB 00:03 2024-03-26T05:13:52.938 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:13:52.959 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:13:52.959 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:13:53.342 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:13:53.343 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:13:53.682 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T05:13:53.683 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T05:13:53.684 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T05:13:53.685 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T05:13:53.685 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T05:13:53.685 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T05:13:53.685 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T05:13:53.685 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T05:13:53.914 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:13:54.013 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T05:13:54.049 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T05:13:54.085 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T05:13:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T05:13:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T05:13:54.889 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T05:13:54.889 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:54.889 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-03-26T05:13:54.889 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.889 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.889 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:13:54.889 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T05:13:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T05:13:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T05:13:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T05:13:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T05:13:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:54.892 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:13:55.211 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-test 2024-03-26T05:13:55.765 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:28 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:13:55.913 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:13:55.913 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:13:55.913 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:13:55.913 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:13:55.914 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:13:55.915 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2024-03-26T05:13:55.915 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:13:55.916 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 48 M 2024-03-26T05:13:55.916 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 205 M 2024-03-26T05:13:55.916 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:13:56.126 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-26T05:13:56.184 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-26T05:13:56.218 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-26T05:13:56.251 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 589 kB/s | 163 kB 00:00 2024-03-26T05:13:56.410 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 975 kB/s | 187 kB 00:00 2024-03-26T05:13:58.632 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T05:13:58.715 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T05:13:58.752 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-26T05:13:58.755 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:13:58.755 INFO:teuthology.orchestra.run.smithi164.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-26T05:13:58.770 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:13:58.792 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:13:58.792 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:13:59.175 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:13:59.175 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:13:59.762 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:13:59.853 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T05:13:59.903 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T05:13:59.934 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T05:13:59.977 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T05:14:00.005 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T05:14:01.153 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T05:14:01.154 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T05:14:01.154 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T05:14:01.154 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T05:14:01.154 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T05:14:01.154 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T05:14:01.587 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T05:14:01.587 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:01.587 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:01.587 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:01.587 INFO:teuthology.orchestra.run.smithi136.stdout: jq-1.6-16.el9.x86_64 2024-03-26T05:14:01.587 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:01.587 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T05:14:01.587 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T05:14:01.588 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T05:14:01.588 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:01.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph 2024-03-26T05:14:02.474 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:42 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:02.627 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:02.629 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-26T05:14:02.629 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:14:02.629 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-26T05:14:02.629 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T05:14:02.630 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T05:14:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T05:14:02.632 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T05:14:02.632 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T05:14:02.632 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T05:14:02.632 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T05:14:02.632 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T05:14:02.632 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T05:14:02.632 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T05:14:02.632 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T05:14:02.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T05:14:02.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T05:14:02.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T05:14:02.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T05:14:02.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T05:14:02.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T05:14:02.634 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T05:14:02.634 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T05:14:02.634 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:02.634 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:02.634 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-26T05:14:02.634 INFO:teuthology.orchestra.run.smithi136.stdout:Install 36 Packages 2024-03-26T05:14:02.635 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:02.635 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 30 M 2024-03-26T05:14:02.635 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 107 M 2024-03-26T05:14:02.635 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:03.240 INFO:teuthology.orchestra.run.smithi136.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-03-26T05:14:03.441 INFO:teuthology.orchestra.run.smithi136.stdout:(2/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 4.1 MB/s | 1.5 MB 00:00 2024-03-26T05:14:04.199 INFO:teuthology.orchestra.run.smithi136.stdout:(3/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86_ 4.9 MB/s | 4.7 MB 00:00 2024-03-26T05:14:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:(4/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 2.0 MB/s | 2.3 MB 00:01 2024-03-26T05:14:04.417 INFO:teuthology.orchestra.run.smithi136.stdout:(5/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86_ 17 MB/s | 17 MB 00:00 2024-03-26T05:14:04.467 INFO:teuthology.orchestra.run.smithi136.stdout:(6/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 930 kB/s | 248 kB 00:00 2024-03-26T05:14:04.546 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T05:14:04.550 INFO:teuthology.orchestra.run.smithi136.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 622 kB/s | 187 kB 00:00 2024-03-26T05:14:04.576 INFO:teuthology.orchestra.run.smithi136.stdout:(8/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2024-03-26T05:14:04.592 INFO:teuthology.orchestra.run.smithi136.stdout:(9/36): python3-markupsafe-1.1.1-12.el9.x86_64. 313 kB/s | 32 kB 00:00 2024-03-26T05:14:04.610 INFO:teuthology.orchestra.run.smithi136.stdout:(10/36): python3-pytz-2021.1-5.el9.noarch.rpm 806 kB/s | 47 kB 00:00 2024-03-26T05:14:04.622 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T05:14:04.626 INFO:teuthology.orchestra.run.smithi136.stdout:(11/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-26T05:14:04.652 INFO:teuthology.orchestra.run.smithi136.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-26T05:14:04.685 INFO:teuthology.orchestra.run.smithi136.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-26T05:14:04.711 INFO:teuthology.orchestra.run.smithi136.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-26T05:14:04.736 INFO:teuthology.orchestra.run.smithi136.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-26T05:14:04.761 INFO:teuthology.orchestra.run.smithi136.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-26T05:14:04.820 INFO:teuthology.orchestra.run.smithi136.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-26T05:14:04.962 INFO:teuthology.orchestra.run.smithi136.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 755 kB/s | 151 kB 00:00 2024-03-26T05:14:04.979 INFO:teuthology.orchestra.run.smithi136.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 273 kB/s | 43 kB 00:00 2024-03-26T05:14:05.029 INFO:teuthology.orchestra.run.smithi136.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-26T05:14:05.063 INFO:teuthology.orchestra.run.smithi136.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-03-26T05:14:05.080 INFO:teuthology.orchestra.run.smithi136.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-26T05:14:05.105 INFO:teuthology.orchestra.run.smithi136.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-03-26T05:14:05.130 INFO:teuthology.orchestra.run.smithi136.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-03-26T05:14:05.156 INFO:teuthology.orchestra.run.smithi136.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-03-26T05:14:05.181 INFO:teuthology.orchestra.run.smithi136.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-03-26T05:14:05.206 INFO:teuthology.orchestra.run.smithi136.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-03-26T05:14:05.240 INFO:teuthology.orchestra.run.smithi136.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-26T05:14:05.265 INFO:teuthology.orchestra.run.smithi136.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-03-26T05:14:05.299 INFO:teuthology.orchestra.run.smithi136.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-26T05:14:05.325 INFO:teuthology.orchestra.run.smithi136.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-26T05:14:05.350 INFO:teuthology.orchestra.run.smithi136.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-26T05:14:05.375 INFO:teuthology.orchestra.run.smithi136.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-26T05:14:05.409 INFO:teuthology.orchestra.run.smithi136.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-26T05:14:05.426 INFO:teuthology.orchestra.run.smithi136.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 404 kB/s | 20 kB 00:00 2024-03-26T05:14:05.467 INFO:teuthology.orchestra.run.smithi136.stdout:(36/36): python3-requests-2.25.1-8.el9.noarch.r 165 kB/s | 125 kB 00:00 2024-03-26T05:14:05.473 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:05.473 INFO:teuthology.orchestra.run.smithi136.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-26T05:14:05.495 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:05.534 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:05.535 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:05.960 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:05.961 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:07.235 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T05:14:07.236 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T05:14:07.236 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T05:14:07.236 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T05:14:07.236 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T05:14:07.236 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T05:14:07.302 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:07.370 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T05:14:07.402 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T05:14:07.430 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T05:14:07.500 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T05:14:07.546 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T05:14:07.579 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T05:14:07.579 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout: jq-1.6-16.el9.x86_64 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:07.580 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:07.669 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T05:14:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T05:14:07.790 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T05:14:07.869 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T05:14:07.924 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T05:14:07.928 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph 2024-03-26T05:14:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T05:14:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T05:14:08.078 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T05:14:08.266 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T05:14:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T05:14:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T05:14:08.399 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T05:14:08.438 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T05:14:08.493 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:41 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:08.513 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T05:14:08.548 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T05:14:08.582 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T05:14:08.622 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T05:14:08.645 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:08.647 INFO:teuthology.orchestra.run.smithi164.stdout:========================================================================================= 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout:========================================================================================= 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T05:14:08.648 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T05:14:08.649 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T05:14:08.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T05:14:08.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T05:14:08.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T05:14:08.651 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:08.651 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:08.651 INFO:teuthology.orchestra.run.smithi164.stdout:========================================================================================= 2024-03-26T05:14:08.651 INFO:teuthology.orchestra.run.smithi164.stdout:Install 36 Packages 2024-03-26T05:14:08.651 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:08.651 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 30 M 2024-03-26T05:14:08.652 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 107 M 2024-03-26T05:14:08.652 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:08.657 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T05:14:08.683 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T05:14:08.772 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T05:14:08.814 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T05:14:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T05:14:09.038 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T05:14:09.166 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T05:14:09.273 INFO:teuthology.orchestra.run.smithi164.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 44 kB/s | 6.6 kB 00:00 2024-03-26T05:14:09.331 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T05:14:09.362 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T05:14:09.387 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T05:14:09.387 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:09.387 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T05:14:09.387 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T05:14:09.387 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T05:14:09.387 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:09.665 INFO:teuthology.orchestra.run.smithi164.stdout:(2/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86_ 12 MB/s | 4.7 MB 00:00 2024-03-26T05:14:09.915 INFO:teuthology.orchestra.run.smithi164.stdout:(3/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 2.9 MB/s | 2.3 MB 00:00 2024-03-26T05:14:09.958 INFO:teuthology.orchestra.run.smithi164.stdout:(4/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2024-03-26T05:14:10.025 INFO:teuthology.orchestra.run.smithi164.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 2.2 MB/s | 248 kB 00:00 2024-03-26T05:14:10.116 INFO:teuthology.orchestra.run.smithi164.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-26T05:14:10.142 INFO:teuthology.orchestra.run.smithi164.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-26T05:14:10.158 INFO:teuthology.orchestra.run.smithi164.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 757 kB/s | 32 kB 00:00 2024-03-26T05:14:10.176 INFO:teuthology.orchestra.run.smithi164.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-26T05:14:10.194 INFO:teuthology.orchestra.run.smithi164.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-26T05:14:10.226 INFO:teuthology.orchestra.run.smithi164.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T05:14:10.293 INFO:teuthology.orchestra.run.smithi164.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-26T05:14:10.318 INFO:teuthology.orchestra.run.smithi164.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-26T05:14:10.344 INFO:teuthology.orchestra.run.smithi164.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T05:14:10.369 INFO:teuthology.orchestra.run.smithi164.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-26T05:14:10.536 INFO:teuthology.orchestra.run.smithi164.stdout:(16/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86 19 MB/s | 17 MB 00:00 2024-03-26T05:14:10.595 INFO:teuthology.orchestra.run.smithi164.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-26T05:14:10.703 INFO:teuthology.orchestra.run.smithi164.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 348 kB/s | 125 kB 00:00 2024-03-26T05:14:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T05:14:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T05:14:10.873 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:10.873 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T05:14:10.873 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T05:14:10.873 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T05:14:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:10.912 INFO:teuthology.orchestra.run.smithi164.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-03-26T05:14:11.095 INFO:teuthology.orchestra.run.smithi164.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 270 kB/s | 151 kB 00:00 2024-03-26T05:14:11.196 INFO:teuthology.orchestra.run.smithi164.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 349 kB/s | 172 kB 00:00 2024-03-26T05:14:11.221 INFO:teuthology.orchestra.run.smithi164.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-03-26T05:14:11.304 INFO:teuthology.orchestra.run.smithi164.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-26T05:14:11.321 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T05:14:11.338 INFO:teuthology.orchestra.run.smithi164.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 841 kB/s | 358 kB 00:00 2024-03-26T05:14:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T05:14:11.344 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:11.344 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T05:14:11.344 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T05:14:11.344 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T05:14:11.344 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:11.363 INFO:teuthology.orchestra.run.smithi164.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2024-03-26T05:14:11.397 INFO:teuthology.orchestra.run.smithi164.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-26T05:14:11.464 INFO:teuthology.orchestra.run.smithi164.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-26T05:14:11.489 INFO:teuthology.orchestra.run.smithi164.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 369 kB/s | 46 kB 00:00 2024-03-26T05:14:11.514 INFO:teuthology.orchestra.run.smithi164.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 672 kB/s | 79 kB 00:00 2024-03-26T05:14:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T05:14:11.587 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T05:14:11.587 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:11.587 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T05:14:11.587 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T05:14:11.587 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T05:14:11.587 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:11.614 INFO:teuthology.orchestra.run.smithi164.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 900 kB/s | 90 kB 00:00 2024-03-26T05:14:11.631 INFO:teuthology.orchestra.run.smithi164.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 115 kB/s | 16 kB 00:00 2024-03-26T05:14:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T05:14:11.738 INFO:teuthology.orchestra.run.smithi164.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-26T05:14:11.738 INFO:teuthology.orchestra.run.smithi164.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-03-26T05:14:11.791 INFO:teuthology.orchestra.run.smithi164.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-03-26T05:14:11.824 INFO:teuthology.orchestra.run.smithi164.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-26T05:14:11.933 INFO:teuthology.orchestra.run.smithi164.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.5 MB/s | 427 kB 00:00 2024-03-26T05:14:11.935 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:11.935 INFO:teuthology.orchestra.run.smithi164.stdout:Total 9.1 MB/s | 30 MB 00:03 2024-03-26T05:14:11.960 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:12.002 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:12.003 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:12.405 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:12.405 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:13.421 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T05:14:13.422 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T05:14:13.423 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T05:14:13.424 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T05:14:13.758 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:13.811 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T05:14:13.843 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T05:14:13.872 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T05:14:13.941 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T05:14:13.995 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T05:14:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T05:14:14.120 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T05:14:14.197 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T05:14:14.239 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T05:14:14.311 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T05:14:14.363 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T05:14:14.363 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:14.363 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:14.363 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T05:14:14.364 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T05:14:14.365 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T05:14:14.366 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T05:14:14.366 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T05:14:14.366 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T05:14:14.366 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T05:14:14.366 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T05:14:14.366 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T05:14:14.366 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:14.366 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:14.373 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T05:14:14.420 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T05:14:14.477 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T05:14:14.535 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T05:14:14.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base 2024-03-26T05:14:14.724 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T05:14:14.775 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T05:14:14.826 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T05:14:14.864 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T05:14:14.894 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T05:14:14.971 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T05:14:15.005 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T05:14:15.038 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T05:14:15.079 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T05:14:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T05:14:15.149 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T05:14:15.237 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T05:14:15.271 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T05:14:15.271 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:15.343 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T05:14:15.372 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:14:15.425 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:15.426 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:14:15.426 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T05:14:15.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install cephadm 2024-03-26T05:14:15.615 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T05:14:15.780 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T05:14:15.810 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T05:14:15.835 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T05:14:15.835 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:15.835 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T05:14:15.835 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T05:14:15.835 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T05:14:15.836 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:16.051 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:16.201 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:16.201 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:16.201 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 748 k 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 754 k 2024-03-26T05:14:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:16.494 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 2.5 MB/s | 748 kB 00:00 2024-03-26T05:14:16.495 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:16.495 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.5 MB/s | 748 kB 00:00 2024-03-26T05:14:16.495 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:16.501 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:16.501 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:16.506 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:16.507 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:16.583 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T05:14:17.142 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T05:14:17.269 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T05:14:17.295 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T05:14:17.295 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:17.295 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T05:14:17.295 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T05:14:17.295 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T05:14:17.295 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T05:14:17.737 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T05:14:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T05:14:17.760 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:17.760 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T05:14:17.760 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T05:14:17.760 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T05:14:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:17.767 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T05:14:17.768 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:17.768 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:17.768 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:17.768 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:17.768 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:17.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T05:14:17.979 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T05:14:18.002 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T05:14:18.002 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:18.002 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T05:14:18.002 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T05:14:18.003 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T05:14:18.003 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:18.068 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T05:14:18.534 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:18.688 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:18.688 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:18.688 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-26T05:14:18.688 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:18.688 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:18.688 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T05:14:18.688 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:18.689 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:18.689 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:18.689 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-26T05:14:18.689 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:18.689 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 145 k 2024-03-26T05:14:18.689 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 436 k 2024-03-26T05:14:18.689 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:19.219 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 274 kB/s | 145 kB 00:00 2024-03-26T05:14:19.219 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:19.220 INFO:teuthology.orchestra.run.smithi136.stdout:Total 273 kB/s | 145 kB 00:00 2024-03-26T05:14:19.220 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:19.227 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:19.227 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:19.281 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:19.281 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:19.422 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:19.495 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T05:14:19.519 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T05:14:19.520 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:19.520 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T05:14:19.520 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:19.827 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T05:14:19.827 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T05:14:19.827 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T05:14:19.827 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T05:14:19.828 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T05:14:19.829 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T05:14:19.830 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T05:14:19.830 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T05:14:20.553 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T05:14:20.553 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:20.553 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:20.553 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:20.553 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:20.553 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T05:14:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:20.708 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T05:14:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T05:14:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T05:14:20.711 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T05:14:20.711 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T05:14:20.711 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T05:14:20.711 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T05:14:20.711 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T05:14:20.711 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:20.711 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:20.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr 2024-03-26T05:14:21.039 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-base 2024-03-26T05:14:21.383 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:21.481 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:14:21.533 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:21.534 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:14:21.534 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:21.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T05:14:21.606 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:21.704 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:14:21.755 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:21.756 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:14:21.756 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:21.819 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install cephadm 2024-03-26T05:14:22.158 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:22.309 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:22.310 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-26T05:14:22.310 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:14:22.310 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T05:14:22.311 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:22.312 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:22.312 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-26T05:14:22.312 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-03-26T05:14:22.312 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:22.312 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 6.9 M 2024-03-26T05:14:22.312 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-03-26T05:14:22.312 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:22.374 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:22.518 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:22.519 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:22.519 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:14:22.519 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:22.519 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:22.519 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T05:14:22.520 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:22.520 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:22.520 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:22.520 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-26T05:14:22.520 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:22.520 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 748 k 2024-03-26T05:14:22.520 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 754 k 2024-03-26T05:14:22.520 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:22.715 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 81 kB/s | 15 kB 00:00 2024-03-26T05:14:22.807 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1775.g5e 112 kB/s | 31 kB 00:00 2024-03-26T05:14:22.858 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-26T05:14:22.891 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-26T05:14:22.942 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 4.3 MB/s | 1.8 MB 00:00 2024-03-26T05:14:22.947 INFO:teuthology.orchestra.run.smithi164.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-26T05:14:22.947 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:22.948 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-26T05:14:22.948 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:22.953 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:22.953 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:22.959 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:22.959 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:23.029 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:23.100 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 144 kB/s | 19 kB 00:00 2024-03-26T05:14:23.175 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-03-26T05:14:23.267 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-26T05:14:23.301 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-26T05:14:23.326 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 527 kB/s | 31 kB 00:00 2024-03-26T05:14:23.351 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-26T05:14:23.455 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T05:14:23.544 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T05:14:23.577 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.2 MB/s | 863 kB 00:00 2024-03-26T05:14:23.581 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:23.581 INFO:teuthology.orchestra.run.smithi136.stdout:Total 5.4 MB/s | 6.9 MB 00:01 2024-03-26T05:14:23.599 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:23.616 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:23.616 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:23.796 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:23.796 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:23.970 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T05:14:24.263 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T05:14:24.263 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:24.263 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:24.263 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:24.263 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:24.263 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:24.330 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:24.410 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T05:14:24.454 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T05:14:24.463 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T05:14:24.500 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T05:14:24.539 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T05:14:24.644 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T05:14:24.774 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T05:14:24.868 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T05:14:24.920 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T05:14:24.948 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T05:14:24.980 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T05:14:25.039 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:25.188 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:25.188 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:25.188 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-26T05:14:25.188 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:25.189 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:25.189 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T05:14:25.189 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:25.189 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:25.189 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:25.189 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-26T05:14:25.189 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:25.189 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 145 k 2024-03-26T05:14:25.190 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 436 k 2024-03-26T05:14:25.190 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:25.224 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T05:14:25.255 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T05:14:25.537 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 418 kB/s | 145 kB 00:00 2024-03-26T05:14:25.538 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:25.538 INFO:teuthology.orchestra.run.smithi164.stdout:Total 416 kB/s | 145 kB 00:00 2024-03-26T05:14:25.538 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:25.546 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:25.546 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:25.598 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:25.599 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:25.734 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:25.788 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T05:14:25.812 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T05:14:25.812 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:25.812 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T05:14:25.812 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T05:14:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T05:14:26.087 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T05:14:26.451 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T05:14:26.452 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T05:14:26.452 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T05:14:26.452 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T05:14:26.452 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:26.452 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:26.710 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T05:14:26.836 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T05:14:26.836 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:26.836 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:26.836 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:26.837 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:26.837 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:27.075 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr 2024-03-26T05:14:27.268 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:27.417 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:27.418 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-26T05:14:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:14:27.418 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-26T05:14:27.418 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T05:14:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:27.420 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:27.420 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-26T05:14:27.420 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-03-26T05:14:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:27.420 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 37 M 2024-03-26T05:14:27.420 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 209 M 2024-03-26T05:14:27.420 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:27.645 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:27.734 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 233 kB/s | 32 kB 00:00 2024-03-26T05:14:27.742 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:14:27.776 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 425 kB/s | 17 kB 00:00 2024-03-26T05:14:27.794 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:27.795 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:14:27.795 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:27.843 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-26T05:14:27.863 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T05:14:27.868 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 380 kB/s | 34 kB 00:00 2024-03-26T05:14:28.052 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-26T05:14:28.119 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-26T05:14:28.144 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T05:14:28.311 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2024-03-26T05:14:28.353 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.9 MB/s | 195 kB 00:00 2024-03-26T05:14:28.414 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:28.520 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-26T05:14:28.564 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:28.565 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2024-03-26T05:14:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:14:28.565 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T05:14:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T05:14:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2024-03-26T05:14:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:Install 12 Packages 2024-03-26T05:14:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 6.9 M 2024-03-26T05:14:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 26 M 2024-03-26T05:14:28.568 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:28.587 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-26T05:14:28.900 INFO:teuthology.orchestra.run.smithi164.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 78 kB/s | 15 kB 00:00 2024-03-26T05:14:29.000 INFO:teuthology.orchestra.run.smithi164.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1775.g5e 106 kB/s | 31 kB 00:00 2024-03-26T05:14:29.067 INFO:teuthology.orchestra.run.smithi164.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 4.9 MB/s | 1.8 MB 00:00 2024-03-26T05:14:29.109 INFO:teuthology.orchestra.run.smithi164.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-26T05:14:29.134 INFO:teuthology.orchestra.run.smithi164.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-26T05:14:29.301 INFO:teuthology.orchestra.run.smithi164.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 117 kB/s | 19 kB 00:00 2024-03-26T05:14:29.321 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.3 MB/s | 7.4 MB 00:01 2024-03-26T05:14:29.323 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:29.323 INFO:teuthology.orchestra.run.smithi136.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-26T05:14:29.363 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:29.387 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:29.387 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:29.426 INFO:teuthology.orchestra.run.smithi164.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-03-26T05:14:29.468 INFO:teuthology.orchestra.run.smithi164.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-26T05:14:29.535 INFO:teuthology.orchestra.run.smithi164.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 710 kB/s | 31 kB 00:00 2024-03-26T05:14:29.627 INFO:teuthology.orchestra.run.smithi164.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-26T05:14:29.686 INFO:teuthology.orchestra.run.smithi164.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 558 kB/s | 144 kB 00:00 2024-03-26T05:14:29.748 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:29.749 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:30.036 INFO:teuthology.orchestra.run.smithi164.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.8 MB/s | 2.0 MB 00:00 2024-03-26T05:14:30.041 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:30.041 INFO:teuthology.orchestra.run.smithi164.stdout:Total 4.7 MB/s | 6.9 MB 00:01 2024-03-26T05:14:30.059 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:30.076 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:30.076 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:30.256 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:30.257 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T05:14:30.559 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T05:14:30.651 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T05:14:30.703 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T05:14:30.767 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T05:14:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T05:14:30.882 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T05:14:30.911 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T05:14:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T05:14:30.957 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T05:14:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T05:14:31.068 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T05:14:31.191 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T05:14:31.285 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T05:14:31.328 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T05:14:31.357 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T05:14:31.388 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T05:14:31.629 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T05:14:31.631 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T05:14:31.672 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T05:14:31.740 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T05:14:32.500 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T05:14:32.501 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T05:14:32.501 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T05:14:32.501 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T05:14:32.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T05:14:32.852 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:32.852 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:32.853 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:32.982 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T05:14:33.106 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T05:14:33.672 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:06 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:33.827 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:33.828 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T05:14:33.829 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout:Install 12 Packages 2024-03-26T05:14:33.830 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:33.831 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 37 M 2024-03-26T05:14:33.831 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 209 M 2024-03-26T05:14:33.831 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:34.013 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T05:14:34.045 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T05:14:34.127 INFO:teuthology.orchestra.run.smithi164.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 235 kB/s | 32 kB 00:00 2024-03-26T05:14:34.160 INFO:teuthology.orchestra.run.smithi164.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 540 kB/s | 17 kB 00:00 2024-03-26T05:14:34.202 INFO:teuthology.orchestra.run.smithi164.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 848 kB/s | 34 kB 00:00 2024-03-26T05:14:34.253 INFO:teuthology.orchestra.run.smithi164.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-26T05:14:34.428 INFO:teuthology.orchestra.run.smithi164.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-26T05:14:34.495 INFO:teuthology.orchestra.run.smithi164.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-26T05:14:34.520 INFO:teuthology.orchestra.run.smithi164.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T05:14:34.696 INFO:teuthology.orchestra.run.smithi164.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-26T05:14:34.838 INFO:teuthology.orchestra.run.smithi164.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-26T05:14:34.871 INFO:teuthology.orchestra.run.smithi164.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-26T05:14:35.063 INFO:teuthology.orchestra.run.smithi164.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-26T05:14:35.614 INFO:teuthology.orchestra.run.smithi164.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.6 MB/s | 7.4 MB 00:01 2024-03-26T05:14:35.616 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:35.616 INFO:teuthology.orchestra.run.smithi164.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-26T05:14:35.657 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:35.682 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:35.682 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:36.032 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:36.032 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T05:14:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T05:14:36.732 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:36.803 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T05:14:36.842 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T05:14:36.937 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T05:14:36.987 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T05:14:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T05:14:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:37.134 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:37.135 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:37.166 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T05:14:37.195 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T05:14:37.272 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T05:14:37.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-rook 2024-03-26T05:14:37.929 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T05:14:38.042 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T05:14:38.095 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:18 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:38.246 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:38.247 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-26T05:14:38.247 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:14:38.247 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-26T05:14:38.247 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:38.247 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T05:14:38.247 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:14:38.247 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T05:14:38.247 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout:Install 13 Packages 2024-03-26T05:14:38.248 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:38.249 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 2.0 M 2024-03-26T05:14:38.249 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-03-26T05:14:38.249 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:38.738 INFO:teuthology.orchestra.run.smithi136.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 189 kB/s | 24 kB 00:00 2024-03-26T05:14:38.763 INFO:teuthology.orchestra.run.smithi136.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-26T05:14:38.814 INFO:teuthology.orchestra.run.smithi136.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-26T05:14:38.839 INFO:teuthology.orchestra.run.smithi136.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-26T05:14:38.883 INFO:teuthology.orchestra.run.smithi136.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.0 MB/s | 211 kB 00:00 2024-03-26T05:14:38.900 INFO:teuthology.orchestra.run.smithi136.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 706 kB/s | 43 kB 00:00 2024-03-26T05:14:38.917 INFO:teuthology.orchestra.run.smithi136.stdout:(7/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 160 kB/s | 49 kB 00:00 2024-03-26T05:14:39.159 INFO:teuthology.orchestra.run.smithi136.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-26T05:14:39.184 INFO:teuthology.orchestra.run.smithi136.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-26T05:14:39.296 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T05:14:39.434 INFO:teuthology.orchestra.run.smithi136.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 392 kB/s | 203 kB 00:00 2024-03-26T05:14:39.526 INFO:teuthology.orchestra.run.smithi136.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-03-26T05:14:39.810 INFO:teuthology.orchestra.run.smithi136.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 239 kB/s | 90 kB 00:00 2024-03-26T05:14:39.843 INFO:teuthology.orchestra.run.smithi136.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-26T05:14:39.846 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:39.846 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-03-26T05:14:39.847 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:39.860 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:39.860 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:39.951 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:39.951 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:40.320 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T05:14:40.345 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T05:14:40.468 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T05:14:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T05:14:40.760 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T05:14:40.796 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T05:14:40.838 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T05:14:40.932 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T05:14:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T05:14:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T05:14:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T05:14:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T05:14:41.920 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T05:14:41.985 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T05:14:42.015 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T05:14:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T05:14:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T05:14:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T05:14:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T05:14:43.120 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T05:14:43.198 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T05:14:43.198 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:43.198 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:43.198 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:43.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:43.199 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:43.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T05:14:43.547 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T05:14:43.548 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T05:14:43.548 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T05:14:43.548 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T05:14:43.548 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T05:14:43.548 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:43.548 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:43.935 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-rook 2024-03-26T05:14:44.018 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:24 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:44.168 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:44.169 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:44.170 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-26T05:14:44.170 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-03-26T05:14:44.170 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:44.170 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 6.8 M 2024-03-26T05:14:44.170 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 33 M 2024-03-26T05:14:44.170 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:44.511 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:17 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:44.647 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-26T05:14:44.662 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T05:14:44.663 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout:Install 13 Packages 2024-03-26T05:14:44.664 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:44.665 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 2.0 M 2024-03-26T05:14:44.665 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 26 M 2024-03-26T05:14:44.665 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 613 kB/s | 138 kB 00:00 2024-03-26T05:14:44.814 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 577 kB/s | 58 kB 00:00 2024-03-26T05:14:44.855 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-26T05:14:44.880 INFO:teuthology.orchestra.run.smithi164.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 145 kB/s | 24 kB 00:00 2024-03-26T05:14:44.897 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-03-26T05:14:44.904 INFO:teuthology.orchestra.run.smithi164.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 93 kB/s | 18 kB 00:00 2024-03-26T05:14:44.930 INFO:teuthology.orchestra.run.smithi164.stdout:(3/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 225 kB/s | 49 kB 00:00 2024-03-26T05:14:44.997 INFO:teuthology.orchestra.run.smithi164.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 174 kB 00:00 2024-03-26T05:14:45.031 INFO:teuthology.orchestra.run.smithi164.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-03-26T05:14:45.064 INFO:teuthology.orchestra.run.smithi164.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 211 kB 00:00 2024-03-26T05:14:45.090 INFO:teuthology.orchestra.run.smithi164.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 463 kB/s | 43 kB 00:00 2024-03-26T05:14:45.131 INFO:teuthology.orchestra.run.smithi164.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-03-26T05:14:45.157 INFO:teuthology.orchestra.run.smithi164.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 163 kB/s | 15 kB 00:00 2024-03-26T05:14:45.224 INFO:teuthology.orchestra.run.smithi164.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 887 kB/s | 59 kB 00:00 2024-03-26T05:14:45.249 INFO:teuthology.orchestra.run.smithi164.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-03-26T05:14:45.283 INFO:teuthology.orchestra.run.smithi164.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-26T05:14:45.316 INFO:teuthology.orchestra.run.smithi164.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-03-26T05:14:45.318 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:45.319 INFO:teuthology.orchestra.run.smithi164.stdout:Total 3.1 MB/s | 2.0 MB 00:00 2024-03-26T05:14:45.319 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:45.331 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.9 MB/s | 5.8 MB 00:00 2024-03-26T05:14:45.332 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:45.332 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:45.332 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:45.333 INFO:teuthology.orchestra.run.smithi136.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-26T05:14:45.345 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:45.355 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:45.355 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:45.422 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:45.422 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:45.444 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:45.445 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:45.915 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T05:14:45.930 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T05:14:46.104 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T05:14:46.160 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T05:14:46.223 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T05:14:46.260 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T05:14:46.301 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T05:14:46.396 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T05:14:46.488 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T05:14:46.496 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T05:14:46.545 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T05:14:46.568 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T05:14:46.638 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T05:14:46.662 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T05:14:47.284 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T05:14:47.342 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T05:14:47.376 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T05:14:47.424 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T05:14:47.446 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T05:14:47.453 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T05:14:47.453 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T05:14:47.453 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T05:14:47.453 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T05:14:47.453 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T05:14:47.453 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T05:14:47.778 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T05:14:47.778 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:47.778 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:47.779 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:47.779 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T05:14:47.779 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T05:14:47.779 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T05:14:47.779 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T05:14:47.779 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T05:14:47.779 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:47.779 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:48.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse 2024-03-26T05:14:48.221 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T05:14:48.222 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T05:14:48.223 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T05:14:48.223 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T05:14:48.593 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T05:14:48.593 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T05:14:48.594 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T05:14:48.595 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T05:14:48.595 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:48.595 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:48.600 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:28 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:48.748 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:48.749 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:48.750 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:48.750 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-26T05:14:48.750 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:48.750 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 967 k 2024-03-26T05:14:48.750 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.8 M 2024-03-26T05:14:48.750 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:48.849 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T05:14:48.875 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 632 kB/s | 79 kB 00:00 2024-03-26T05:14:49.084 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 2.6 MB/s | 888 kB 00:00 2024-03-26T05:14:49.085 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:49.085 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.8 MB/s | 967 kB 00:00 2024-03-26T05:14:49.085 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:49.105 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:49.106 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:49.180 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:49.181 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:49.355 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:49.405 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:22 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:49.498 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T05:14:49.554 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:49.554 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-26T05:14:49.554 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:14:49.554 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-26T05:14:49.554 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2024-03-26T05:14:49.555 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2024-03-26T05:14:49.556 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:49.556 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 6.8 M 2024-03-26T05:14:49.556 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 33 M 2024-03-26T05:14:49.556 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:49.575 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T05:14:50.210 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 609 kB/s | 138 kB 00:00 2024-03-26T05:14:50.285 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-26T05:14:50.301 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T05:14:50.301 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T05:14:50.311 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 757 kB/s | 248 kB 00:00 2024-03-26T05:14:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T05:14:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:50.636 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T05:14:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:50.636 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:50.853 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 102 kB/s | 58 kB 00:00 2024-03-26T05:14:50.878 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 90 kB/s | 51 kB 00:00 2024-03-26T05:14:50.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-volume 2024-03-26T05:14:51.386 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 466 kB/s | 548 kB 00:01 2024-03-26T05:14:51.390 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:51.390 INFO:teuthology.orchestra.run.smithi164.stdout:Total 3.7 MB/s | 6.8 MB 00:01 2024-03-26T05:14:51.403 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:51.413 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:51.414 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:51.503 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:51.504 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:51.542 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:31 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:51.692 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-26T05:14:51.693 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:51.694 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 332 k 2024-03-26T05:14:51.694 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 1.5 M 2024-03-26T05:14:51.694 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:51.801 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:51.865 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 402 kB/s | 69 kB 00:00 2024-03-26T05:14:51.969 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T05:14:52.013 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T05:14:52.132 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 602 kB/s | 264 kB 00:00 2024-03-26T05:14:52.133 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:52.133 INFO:teuthology.orchestra.run.smithi136.stdout:Total 755 kB/s | 332 kB 00:00 2024-03-26T05:14:52.133 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:52.142 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:52.142 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:52.177 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:52.177 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:52.327 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:52.513 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T05:14:52.527 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T05:14:52.559 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T05:14:52.575 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T05:14:52.576 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:52.576 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T05:14:52.576 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:52.607 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T05:14:52.668 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T05:14:52.692 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T05:14:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T05:14:53.511 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T05:14:53.512 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T05:14:53.512 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T05:14:53.512 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T05:14:53.512 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T05:14:53.512 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T05:14:53.727 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T05:14:53.728 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:53.728 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:53.728 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:53.728 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T05:14:53.728 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:53.728 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:53.824 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:53.825 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:54.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel 2024-03-26T05:14:54.075 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-fuse 2024-03-26T05:14:54.629 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:34 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:54.636 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:54.779 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 127 k 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 456 k 2024-03-26T05:14:54.780 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:54.786 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:54.787 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:14:54.787 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:54.787 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T05:14:54.787 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:14:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T05:14:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:54.788 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:54.788 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:54.788 INFO:teuthology.orchestra.run.smithi164.stdout:Install 2 Packages 2024-03-26T05:14:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:54.788 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 967 k 2024-03-26T05:14:54.789 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 2.8 M 2024-03-26T05:14:54.789 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:54.922 INFO:teuthology.orchestra.run.smithi164.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 591 kB/s | 79 kB 00:00 2024-03-26T05:14:55.088 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 412 kB/s | 127 kB 00:00 2024-03-26T05:14:55.089 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:55.089 INFO:teuthology.orchestra.run.smithi136.stdout:Total 410 kB/s | 127 kB 00:00 2024-03-26T05:14:55.089 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:55.095 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:55.095 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:55.151 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:55.151 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:55.288 INFO:teuthology.orchestra.run.smithi164.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 1.7 MB/s | 888 kB 00:00 2024-03-26T05:14:55.290 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:55.290 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.9 MB/s | 967 kB 00:00 2024-03-26T05:14:55.290 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:55.306 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:55.311 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:55.311 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:55.385 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:55.385 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:55.412 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:14:55.563 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:55.707 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T05:14:55.771 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T05:14:55.869 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:14:56.177 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:14:56.177 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:56.177 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:56.177 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:56.177 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:56.177 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:56.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2 2024-03-26T05:14:56.509 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T05:14:56.510 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T05:14:56.848 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T05:14:56.848 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:56.849 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:56.849 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T05:14:56.849 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:56.849 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:56.989 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:57.089 INFO:teuthology.orchestra.run.smithi136.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:14:57.136 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-volume 2024-03-26T05:14:57.143 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:57.144 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:14:57.144 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:57.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs-devel 2024-03-26T05:14:57.716 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:14:57.806 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:37 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:57.868 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:14:57.868 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:57.868 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-26T05:14:57.868 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:57.868 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:14:57.868 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T05:14:57.868 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T05:14:57.868 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T05:14:57.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:57.869 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:14:57.869 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:14:57.869 INFO:teuthology.orchestra.run.smithi164.stdout:Install 2 Packages 2024-03-26T05:14:57.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:57.869 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 332 k 2024-03-26T05:14:57.869 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 1.5 M 2024-03-26T05:14:57.869 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:14:57.957 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 31 k 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 140 k 2024-03-26T05:14:57.958 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:14:57.997 INFO:teuthology.orchestra.run.smithi164.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 541 kB/s | 69 kB 00:00 2024-03-26T05:14:58.126 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 186 kB/s | 31 kB 00:00 2024-03-26T05:14:58.126 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:58.127 INFO:teuthology.orchestra.run.smithi136.stdout:Total 185 kB/s | 31 kB 00:00 2024-03-26T05:14:58.127 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:14:58.130 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:14:58.130 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:14:58.154 INFO:teuthology.orchestra.run.smithi164.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 927 kB/s | 264 kB 00:00 2024-03-26T05:14:58.155 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:14:58.156 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-26T05:14:58.156 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:14:58.157 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:14:58.157 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:14:58.164 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:14:58.164 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:14:58.200 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:14:58.201 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:14:58.282 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:14:58.358 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:14:58.390 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:14:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T05:14:58.590 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T05:14:58.606 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T05:14:58.606 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:14:58.606 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T05:14:58.606 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:58.772 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:14:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:14:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:59.044 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:14:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:14:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:14:59.044 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:59.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2 2024-03-26T05:14:59.437 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T05:14:59.780 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:14:59.791 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T05:14:59.791 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:59.791 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:14:59.791 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T05:14:59.791 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T05:14:59.791 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:14:59.792 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:14:59.878 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:14:59.930 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:14:59.931 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:14:59.931 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:14:59.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1 2024-03-26T05:15:00.130 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados-devel 2024-03-26T05:15:00.546 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:15:00.643 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:00.696 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:15:00.697 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:15:00.697 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:15:00.704 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:00.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados 2024-03-26T05:15:00.854 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:00.854 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:00.854 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-26T05:15:00.855 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:00.855 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:15:00.855 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T05:15:00.855 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:00.855 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:15:00.855 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:00.855 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-26T05:15:00.855 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:00.856 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 127 k 2024-03-26T05:15:00.856 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 456 k 2024-03-26T05:15:00.856 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:15:01.086 INFO:teuthology.orchestra.run.smithi164.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 550 kB/s | 127 kB 00:00 2024-03-26T05:15:01.087 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:01.087 INFO:teuthology.orchestra.run.smithi164.stdout:Total 546 kB/s | 127 kB 00:00 2024-03-26T05:15:01.087 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:15:01.093 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:15:01.093 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:15:01.149 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:15:01.150 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:15:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:15:01.323 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:15:01.401 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:01.420 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:01.471 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:15:01.472 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:15:01.472 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:15:01.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw 2024-03-26T05:15:01.844 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:02.097 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:15:02.130 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:02.130 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:02.130 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:15:02.130 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:15:02.131 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:02.131 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:02.194 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:02.246 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:15:02.247 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:15:02.247 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:15:02.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs 2024-03-26T05:15:02.381 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs2 2024-03-26T05:15:02.862 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:15:02.957 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:35 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:02.959 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:03.010 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:15:03.011 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:15:03.011 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:15:03.057 INFO:teuthology.orchestra.run.smithi164.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:03.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd 2024-03-26T05:15:03.112 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:03.113 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:15:03.113 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:03.174 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs-devel 2024-03-26T05:15:03.630 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:15:03.728 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:03.756 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:03.780 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:15:03.781 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-26T05:15:03.781 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:15:03.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse 2024-03-26T05:15:03.922 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:03.922 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:03.922 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 31 k 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 140 k 2024-03-26T05:15:03.923 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:15:04.092 INFO:teuthology.orchestra.run.smithi164.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 186 kB/s | 31 kB 00:00 2024-03-26T05:15:04.092 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:04.093 INFO:teuthology.orchestra.run.smithi164.stdout:Total 184 kB/s | 31 kB 00:00 2024-03-26T05:15:04.093 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:15:04.096 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:15:04.096 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:15:04.126 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:15:04.127 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:15:04.249 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:15:04.374 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:04.405 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:15:04.566 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 85 k 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 231 k 2024-03-26T05:15:04.567 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:15:04.780 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 402 kB/s | 85 kB 00:00 2024-03-26T05:15:04.780 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:04.781 INFO:teuthology.orchestra.run.smithi136.stdout:Total 399 kB/s | 85 kB 00:00 2024-03-26T05:15:04.781 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:15:04.788 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:15:04.788 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:15:04.833 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:04.846 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:15:04.846 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:15:04.989 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:15:05.088 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:05.104 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:05.104 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:05.105 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:15:05.105 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:15:05.105 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:05.105 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:05.290 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados2 2024-03-26T05:15:05.518 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:05.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:05.816 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:05.816 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:15:05.816 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:15:05.816 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:05.816 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:15:05.856 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:05.954 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:06.008 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:06.009 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:15:06.009 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:06.067 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librbd1 2024-03-26T05:15:06.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror 2024-03-26T05:15:06.627 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:06.642 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:15:06.723 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:06.776 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:06.777 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:15:06.777 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:06.793 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-26T05:15:06.794 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:06.795 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.1 M 2024-03-26T05:15:06.795 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 13 M 2024-03-26T05:15:06.795 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:15:06.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rados 2024-03-26T05:15:07.398 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:07.494 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:07.546 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:07.547 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:15:07.547 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:07.599 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rgw 2024-03-26T05:15:08.165 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:08.316 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:08.317 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:15:08.317 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:08.362 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-cephfs 2024-03-26T05:15:08.925 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:09.023 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:09.074 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:09.075 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:15:09.075 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:09.123 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rbd 2024-03-26T05:15:09.625 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 1.1 MB/s | 3.1 MB 00:02 2024-03-26T05:15:09.626 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:09.627 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.1 MB/s | 3.1 MB 00:02 2024-03-26T05:15:09.627 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:15:09.638 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:15:09.638 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:15:09.691 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:09.716 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:15:09.716 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:15:09.789 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T05:15:09.840 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:09.841 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-03-26T05:15:09.841 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:09.899 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-fuse 2024-03-26T05:15:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:15:10.209 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:10.234 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:10.235 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:15:10.235 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T05:15:10.235 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T05:15:10.235 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T05:15:10.235 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:10.465 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:10.614 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 85 k 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 231 k 2024-03-26T05:15:10.615 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:15:10.839 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 382 kB/s | 85 kB 00:00 2024-03-26T05:15:10.839 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:10.839 INFO:teuthology.orchestra.run.smithi164.stdout:Total 380 kB/s | 85 kB 00:00 2024-03-26T05:15:10.840 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:15:10.847 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:15:10.847 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:15:10.902 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:15:10.902 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:15:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:15:11.151 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:11.420 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:11.420 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:11.420 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:15:11.420 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:15:11.420 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:11.420 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:15:11.570 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:11.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd 2024-03-26T05:15:11.898 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:11.898 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:11.898 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:15:11.898 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:15:11.898 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:11.898 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:12.147 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-mirror 2024-03-26T05:15:12.223 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:52 ago on Tue 26 Mar 2024 05:13:20 AM UTC. 2024-03-26T05:15:12.373 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-26T05:15:12.374 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:12.375 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 171 k 2024-03-26T05:15:12.375 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 494 k 2024-03-26T05:15:12.375 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-26T05:15:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:12.879 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:12.880 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:12.880 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-03-26T05:15:12.880 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:12.880 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:15:12.880 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T05:15:12.880 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:12.880 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:15:12.881 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:12.881 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-26T05:15:12.881 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:12.881 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 3.1 M 2024-03-26T05:15:12.881 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 13 M 2024-03-26T05:15:12.881 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:15:12.888 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 333 kB/s | 171 kB 00:00 2024-03-26T05:15:12.888 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:12.889 INFO:teuthology.orchestra.run.smithi136.stdout:Total 332 kB/s | 171 kB 00:00 2024-03-26T05:15:12.889 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-26T05:15:12.897 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-26T05:15:12.897 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-26T05:15:12.953 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-26T05:15:12.953 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-26T05:15:13.098 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-26T05:15:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:13.625 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:13.732 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 3.6 MB/s | 3.1 MB 00:00 2024-03-26T05:15:13.733 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:13.734 INFO:teuthology.orchestra.run.smithi164.stdout:Total 3.6 MB/s | 3.1 MB 00:00 2024-03-26T05:15:13.734 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:15:13.743 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:15:13.743 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:15:13.819 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:15:13.819 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:15:13.923 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:13.923 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:13.923 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-26T05:15:13.923 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:15:13.923 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:13.923 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-26T05:15:14.150 DEBUG:teuthology.parallel:result is None 2024-03-26T05:15:14.247 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:15:14.317 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:14.340 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:14.340 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T05:15:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T05:15:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T05:15:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T05:15:14.341 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:15.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:15.455 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:15:15.455 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:15:15.455 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:15.694 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-nbd 2024-03-26T05:15:16.268 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Mar 2024 05:13:27 AM UTC. 2024-03-26T05:15:16.418 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T05:15:16.419 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:16.419 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-03-26T05:15:16.419 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:16.419 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T05:15:16.419 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T05:15:16.420 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T05:15:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T05:15:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-03-26T05:15:16.420 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 171 k 2024-03-26T05:15:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 494 k 2024-03-26T05:15:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T05:15:16.645 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 758 kB/s | 171 kB 00:00 2024-03-26T05:15:16.646 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:16.646 INFO:teuthology.orchestra.run.smithi164.stdout:Total 753 kB/s | 171 kB 00:00 2024-03-26T05:15:16.647 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T05:15:16.654 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T05:15:16.654 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T05:15:16.710 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T05:15:16.710 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T05:15:16.866 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T05:15:16.978 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:17.421 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:17.727 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T05:15:17.727 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:17.728 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T05:15:17.728 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T05:15:17.728 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:15:17.728 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T05:15:17.974 DEBUG:teuthology.parallel:result is None 2024-03-26T05:15:17.975 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T05:15:18.116 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T05:15:18.141 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T05:15:18.141 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T05:15:18.142 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T05:15:18.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T05:15:18.358 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T05:15:18.379 INFO:teuthology.orchestra.run.smithi164.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T05:15:18.379 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T05:15:18.379 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T05:15:18.381 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T05:15:18.381 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:15:18.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T05:15:18.413 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:15:18.413 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T05:15:18.447 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T05:15:18.447 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:15:18.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T05:15:18.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T05:15:18.553 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:15:18.553 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T05:15:18.580 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T05:15:18.673 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T05:15:18.674 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:15:18.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T05:15:18.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T05:15:18.772 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:15:18.772 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T05:15:18.799 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T05:15:18.866 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T05:15:18.866 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:15:18.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T05:15:18.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T05:15:18.963 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:15:18.963 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T05:15:18.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T05:15:19.058 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T05:15:19.154 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'cephadm_mode': 'cephadm-package'} 2024-03-26T05:15:19.154 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T05:15:19.155 INFO:tasks.cephadm:Cluster fsid is d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:15:19.155 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T05:15:19.155 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.136', 'mon.c': '[v2:172.21.15.136:3301,v1:172.21.15.136:6790]', 'mon.b': '172.21.15.164'} 2024-03-26T05:15:19.155 INFO:tasks.cephadm:First mon is mon.a on smithi136 2024-03-26T05:15:19.155 INFO:tasks.cephadm:First mgr is y 2024-03-26T05:15:19.155 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T05:15:19.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-03-26T05:15:19.184 DEBUG:teuthology.orchestra.run.smithi164:> sudo hostname $(hostname -s) 2024-03-26T05:15:19.221 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T05:15:19.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T05:15:19.227 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T05:15:20.123 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T05:15:20.200 INFO:teuthology.orchestra.run.smithi164.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T05:15:46.490 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-03-26T05:15:46.490 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T05:15:46.490 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T05:15:46.490 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-03-26T05:15:46.490 INFO:teuthology.orchestra.run.smithi136.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T05:15:46.490 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-03-26T05:15:46.490 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-03-26T05:15:46.683 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T05:15:46.683 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T05:15:46.683 INFO:teuthology.orchestra.run.smithi164.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T05:15:46.683 INFO:teuthology.orchestra.run.smithi164.stdout: "repo_digests": [ 2024-03-26T05:15:46.683 INFO:teuthology.orchestra.run.smithi164.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T05:15:46.683 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-03-26T05:15:46.683 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T05:15:46.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-03-26T05:15:46.738 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph 2024-03-26T05:15:46.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-03-26T05:15:46.805 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /etc/ceph 2024-03-26T05:15:46.838 INFO:tasks.cephadm:Writing seed config... 2024-03-26T05:15:46.839 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T05:15:46.839 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T05:15:46.839 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-26T05:15:46.840 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T05:15:46.841 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:15:46.841 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T05:15:46.860 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d69b39a8-eb2f-11ee-95d0-87774f69a715 mon election default strategy = 1 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-26T05:15:46.860 DEBUG:teuthology.orchestra.run.smithi136:mon.a> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T05:15:46.903 DEBUG:teuthology.orchestra.run.smithi136:mgr.y> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T05:15:46.945 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T05:15:46.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.136 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:15:47.135 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-26T05:15:47.135 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', 'd69b39a8-eb2f-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.136', '--skip-admin-label'] 2024-03-26T05:15:47.160 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T05:15:47.161 INFO:teuthology.orchestra.run.smithi136.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T05:15:47.161 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying podman|docker is present... 2024-03-26T05:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T05:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying lvm2 is present... 2024-03-26T05:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying time synchronization is in place... 2024-03-26T05:15:47.192 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T05:15:47.192 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T05:15:47.198 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T05:15:47.198 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T05:15:47.204 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout enabled 2024-03-26T05:15:47.210 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout active 2024-03-26T05:15:47.210 INFO:teuthology.orchestra.run.smithi136.stdout:Unit chronyd.service is enabled and running 2024-03-26T05:15:47.210 INFO:teuthology.orchestra.run.smithi136.stdout:Repeating the final host check... 2024-03-26T05:15:47.234 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T05:15:47.234 INFO:teuthology.orchestra.run.smithi136.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T05:15:47.234 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl is present 2024-03-26T05:15:47.234 INFO:teuthology.orchestra.run.smithi136.stdout:lvcreate is present 2024-03-26T05:15:47.240 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T05:15:47.240 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T05:15:47.246 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T05:15:47.246 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T05:15:47.252 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout enabled 2024-03-26T05:15:47.258 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout active 2024-03-26T05:15:47.259 INFO:teuthology.orchestra.run.smithi136.stdout:Unit chronyd.service is enabled and running 2024-03-26T05:15:47.259 INFO:teuthology.orchestra.run.smithi136.stdout:Host looks OK 2024-03-26T05:15:47.259 INFO:teuthology.orchestra.run.smithi136.stdout:Cluster fsid: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:15:47.259 INFO:teuthology.orchestra.run.smithi136.stdout:Acquiring lock 139677818820880 on /run/cephadm/d69b39a8-eb2f-11ee-95d0-87774f69a715.lock 2024-03-26T05:15:47.259 INFO:teuthology.orchestra.run.smithi136.stdout:Lock 139677818820880 acquired on /run/cephadm/d69b39a8-eb2f-11ee-95d0-87774f69a715.lock 2024-03-26T05:15:47.259 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying IP 172.21.15.136 port 3300 ... 2024-03-26T05:15:47.259 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying IP 172.21.15.136 port 6789 ... 2024-03-26T05:15:47.259 INFO:teuthology.orchestra.run.smithi136.stdout:Base mon IP(s) is [172.21.15.136:3300, 172.21.15.136:6789], mon addrv is [v2:172.21.15.136:3300,v1:172.21.15.136:6789] 2024-03-26T05:15:47.263 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.136 metric 100 2024-03-26T05:15:47.263 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.136 metric 100 2024-03-26T05:15:47.265 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T05:15:47.265 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T05:15:47.265 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-03-26T05:15:47.268 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T05:15:47.268 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T05:15:47.268 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T05:15:47.268 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T05:15:47.268 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6e43/64 scope link 2024-03-26T05:15:47.268 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T05:15:47.269 INFO:teuthology.orchestra.run.smithi136.stdout:Mon IP `172.21.15.136` is in CIDR network `172.21.0.0/20` 2024-03-26T05:15:47.269 INFO:teuthology.orchestra.run.smithi136.stdout:Mon IP `172.21.15.136` is in CIDR network `172.21.0.0/20` 2024-03-26T05:15:47.269 INFO:teuthology.orchestra.run.smithi136.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T05:15:47.269 INFO:teuthology.orchestra.run.smithi136.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T05:15:47.270 INFO:teuthology.orchestra.run.smithi136.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T05:15:47.759 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T05:15:47.759 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T05:15:47.759 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T05:15:47.759 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T05:15:47.759 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T05:15:47.759 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T05:15:47.759 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T05:15:48.689 INFO:teuthology.orchestra.run.smithi136.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T05:15:48.689 INFO:teuthology.orchestra.run.smithi136.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T05:15:48.689 INFO:teuthology.orchestra.run.smithi136.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T05:15:49.586 INFO:teuthology.orchestra.run.smithi136.stdout:stat: stdout 167 167 2024-03-26T05:15:49.586 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial keys... 2024-03-26T05:15:50.607 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQAFWgJm8OpsOBAA8P9ZSfgOxKxXX6I6SEGnbg== 2024-03-26T05:15:51.528 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQAGWgJm4Sy/NxAAAH9ySItvy1yyAVKODvzA+w== 2024-03-26T05:15:52.476 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQAHWgJm/u2LMxAAmk1oAW5o82NL7KluscIm7Q== 2024-03-26T05:15:52.476 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial monmap... 2024-03-26T05:15:53.390 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool for a [v2:172.21.15.136:3300,v1:172.21.15.136:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout:setting min_mon_release = quincy 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: set fsid to d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:15:53.391 INFO:teuthology.orchestra.run.smithi136.stdout:Creating mon... 2024-03-26T05:15:54.295 INFO:teuthology.orchestra.run.smithi136.stdout:create mon.a on 2024-03-26T05:15:54.497 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T05:15:54.636 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T05:15:54.817 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715.target. 2024-03-26T05:15:54.817 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715.target. 2024-03-26T05:15:55.039 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.a 2024-03-26T05:15:55.039 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T05:15:55.195 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715.target.wants/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@.service. 2024-03-26T05:15:55.701 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T05:15:55.702 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-03-26T05:15:55.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:55 smithi136 systemd[1]: Started Ceph mon.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:15:55.710 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T05:15:55.710 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T05:15:55.710 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-03-26T05:15:55.710 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T05:15:55.711 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mon to start... 2024-03-26T05:15:55.711 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mon... 2024-03-26T05:15:55.968 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:55 smithi136 ceph-mon[29174]: mkfs d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:15:55.968 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:55 smithi136 ceph-mon[29174]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T05:15:56.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:56 smithi136 ceph-mon[29174]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T05:15:56.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:56 smithi136 ceph-mon[29174]: monmap e1: 1 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T05:15:56.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:56 smithi136 ceph-mon[29174]: fsmap 2024-03-26T05:15:56.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:56 smithi136 ceph-mon[29174]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T05:15:56.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:56 smithi136 ceph-mon[29174]: mgrmap e1: no daemons active 2024-03-26T05:15:56.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:56 smithi136 ceph-mon[29174]: from='client.? 172.21.15.136:0/4065007827' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T05:15:56.805 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T05:15:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout id: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout services: 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.551964s) 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout data: 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T05:15:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.173+0000 7f6dbe907700 1 Processor -- start 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.173+0000 7f6dbe907700 1 -- start start 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.173+0000 7f6dbe907700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db8106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.173+0000 7f6dbe907700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8106b00 con 0x7f6db8106110 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.174+0000 7f6db7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db8106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.174+0000 7f6db7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db8106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46256/0 (socket says 172.21.15.136:46256) 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.174+0000 7f6db7fff700 1 -- 172.21.15.136:0/3661522514 learned_addr learned my addr 172.21.15.136:0/3661522514 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.174+0000 7f6db7fff700 1 -- 172.21.15.136:0/3661522514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8107310 con 0x7f6db8106110 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.174+0000 7f6db7fff700 1 --2- 172.21.15.136:0/3661522514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db8106530 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6da80098a0 tx=0x7f6da8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adee619742f45bd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.174+0000 7f6db6ffd700 1 -- 172.21.15.136:0/3661522514 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6da8004160 con 0x7f6db8106110 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.174+0000 7f6db6ffd700 1 -- 172.21.15.136:0/3661522514 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da80042c0 con 0x7f6db8106110 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.174+0000 7f6db6ffd700 1 -- 172.21.15.136:0/3661522514 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6da80045a0 con 0x7f6db8106110 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.175+0000 7f6dbe907700 1 -- 172.21.15.136:0/3661522514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 msgr2=0x7f6db8106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:15:56.813 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.175+0000 7f6dbe907700 1 --2- 172.21.15.136:0/3661522514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db8106530 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6da80098a0 tx=0x7f6da8009bb0 comp rx=0 tx=0).stop 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.175+0000 7f6dbe907700 1 -- 172.21.15.136:0/3661522514 shutdown_connections 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.175+0000 7f6dbe907700 1 --2- 172.21.15.136:0/3661522514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db8106530 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.175+0000 7f6dbe907700 1 -- 172.21.15.136:0/3661522514 >> 172.21.15.136:0/3661522514 conn(0x7f6db8101670 msgr2=0x7f6db8103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.175+0000 7f6dbe907700 1 -- 172.21.15.136:0/3661522514 shutdown_connections 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.175+0000 7f6dbe907700 1 -- 172.21.15.136:0/3661522514 wait complete. 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.175+0000 7f6dbe907700 1 Processor -- start 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.176+0000 7f6dbe907700 1 -- start start 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.176+0000 7f6dbe907700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db807bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.176+0000 7f6dbe907700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db807c1b0 con 0x7f6db8106110 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.176+0000 7f6db7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db807bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.176+0000 7f6db7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db807bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46258/0 (socket says 172.21.15.136:46258) 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.176+0000 7f6db7fff700 1 -- 172.21.15.136:0/4065007827 learned_addr learned my addr 172.21.15.136:0/4065007827 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.176+0000 7f6db7fff700 1 -- 172.21.15.136:0/4065007827 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da8009550 con 0x7f6db8106110 2024-03-26T05:15:56.814 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.176+0000 7f6db7fff700 1 --2- 172.21.15.136:0/4065007827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db807bc70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6da8004000 tx=0x7f6da8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.177+0000 7f6db57fa700 1 -- 172.21.15.136:0/4065007827 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6da8011630 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.177+0000 7f6db57fa700 1 -- 172.21.15.136:0/4065007827 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8011790 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.177+0000 7f6db57fa700 1 -- 172.21.15.136:0/4065007827 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6da8011a30 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.177+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db807a3a0 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.177+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db807a840 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.177+0000 7f6db57fa700 1 -- 172.21.15.136:0/4065007827 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6da8022070 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.177+0000 7f6db57fa700 1 -- 172.21.15.136:0/4065007827 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6da801c450 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.178+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db80636b0 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.180+0000 7f6db57fa700 1 -- 172.21.15.136:0/4065007827 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6da8011b90 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.250+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f6db8107680 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.250+0000 7f6db57fa700 1 -- 172.21.15.136:0/4065007827 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f6da8011d30 con 0x7f6db8106110 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.251+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 msgr2=0x7f6db807bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.251+0000 7f6dbe907700 1 --2- 172.21.15.136:0/4065007827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db807bc70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6da8004000 tx=0x7f6da8004750 comp rx=0 tx=0).stop 2024-03-26T05:15:56.815 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.251+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 shutdown_connections 2024-03-26T05:15:56.816 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.251+0000 7f6dbe907700 1 --2- 172.21.15.136:0/4065007827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db8106110 0x7f6db807bc70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:15:56.816 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.251+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 >> 172.21.15.136:0/4065007827 conn(0x7f6db8101670 msgr2=0x7f6db81904c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:15:56.816 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.251+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 shutdown_connections 2024-03-26T05:15:56.816 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:56.251+0000 7f6dbe907700 1 -- 172.21.15.136:0/4065007827 wait complete. 2024-03-26T05:15:56.816 INFO:teuthology.orchestra.run.smithi136.stdout:mon is available 2024-03-26T05:15:56.816 INFO:teuthology.orchestra.run.smithi136.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T05:15:57.808 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:57 smithi136 ceph-mon[29174]: from='client.? 172.21.15.136:0/2875303982' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T05:15:57.809 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:57 smithi136 ceph-mon[29174]: from='client.? 172.21.15.136:0/2875303982' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T05:15:57.817 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:15:57.817 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T05:15:57.817 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout fsid = d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:15:57.817 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.136:3300,v1:172.21.15.136:6789] 2024-03-26T05:15:57.817 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T05:15:57.817 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.197+0000 7f249a563700 1 Processor -- start 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.197+0000 7f249a563700 1 -- start start 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.197+0000 7f249a563700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f2494109010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.197+0000 7f249a563700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2494079640 con 0x7f2494106c20 2024-03-26T05:15:57.818 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.198+0000 7f2493fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f2494109010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.198+0000 7f2493fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f2494109010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46264/0 (socket says 172.21.15.136:46264) 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.198+0000 7f2493fff700 1 -- 172.21.15.136:0/2250135311 learned_addr learned my addr 172.21.15.136:0/2250135311 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.198+0000 7f2493fff700 1 -- 172.21.15.136:0/2250135311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2494079780 con 0x7f2494106c20 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.198+0000 7f2493fff700 1 --2- 172.21.15.136:0/2250135311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f2494109010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f24840098a0 tx=0x7f2484009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e7a6fafd6a79d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.198+0000 7f2492ffd700 1 -- 172.21.15.136:0/2250135311 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2484004160 con 0x7f2494106c20 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.198+0000 7f2492ffd700 1 -- 172.21.15.136:0/2250135311 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24840042c0 con 0x7f2494106c20 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.198+0000 7f2492ffd700 1 -- 172.21.15.136:0/2250135311 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24840045a0 con 0x7f2494106c20 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 -- 172.21.15.136:0/2250135311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 msgr2=0x7f2494109010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 --2- 172.21.15.136:0/2250135311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f2494109010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f24840098a0 tx=0x7f2484009bb0 comp rx=0 tx=0).stop 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 -- 172.21.15.136:0/2250135311 shutdown_connections 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 --2- 172.21.15.136:0/2250135311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f2494109010 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 -- 172.21.15.136:0/2250135311 >> 172.21.15.136:0/2250135311 conn(0x7f2494100590 msgr2=0x7f24941029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:15:57.819 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 -- 172.21.15.136:0/2250135311 shutdown_connections 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 -- 172.21.15.136:0/2250135311 wait complete. 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 Processor -- start 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.199+0000 7f249a563700 1 -- start start 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.200+0000 7f249a563700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f249419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.200+0000 7f249a563700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249419bd90 con 0x7f2494106c20 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.200+0000 7f2493fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f249419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.200+0000 7f2493fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f249419b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46278/0 (socket says 172.21.15.136:46278) 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.200+0000 7f2493fff700 1 -- 172.21.15.136:0/2875303982 learned_addr learned my addr 172.21.15.136:0/2875303982 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:15:57.820 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.200+0000 7f2493fff700 1 -- 172.21.15.136:0/2875303982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2484009550 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.200+0000 7f2493fff700 1 --2- 172.21.15.136:0/2875303982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f249419b850 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2494078a80 tx=0x7f2484004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.200+0000 7f24917fa700 1 -- 172.21.15.136:0/2875303982 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2484011630 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.201+0000 7f24917fa700 1 -- 172.21.15.136:0/2875303982 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2484011790 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.201+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249419bf90 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.201+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249419c430 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.201+0000 7f24917fa700 1 -- 172.21.15.136:0/2875303982 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2484011a30 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.201+0000 7f24917fa700 1 -- 172.21.15.136:0/2875303982 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2484022070 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.201+0000 7f24917fa700 1 -- 172.21.15.136:0/2875303982 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f248401c450 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.202+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2480005120 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.204+0000 7f24917fa700 1 -- 172.21.15.136:0/2875303982 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2484011b90 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.292+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2480002400 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.294+0000 7f24917fa700 1 -- 172.21.15.136:0/2875303982 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+340 (secure 0 0 0) 0x7f2484011de0 con 0x7f2494106c20 2024-03-26T05:15:57.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.295+0000 7f24917fa700 1 -- 172.21.15.136:0/2875303982 <== mon.0 v2:172.21.15.136:3300/0 8 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f248401c610 con 0x7f2494106c20 2024-03-26T05:15:57.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.296+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 msgr2=0x7f249419b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:15:57.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.296+0000 7f249a563700 1 --2- 172.21.15.136:0/2875303982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f249419b850 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2494078a80 tx=0x7f2484004750 comp rx=0 tx=0).stop 2024-03-26T05:15:57.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.296+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 shutdown_connections 2024-03-26T05:15:57.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.296+0000 7f249a563700 1 --2- 172.21.15.136:0/2875303982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2494106c20 0x7f249419b850 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:15:57.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.296+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 >> 172.21.15.136:0/2875303982 conn(0x7f2494100590 msgr2=0x7f2494102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:15:57.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.296+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 shutdown_connections 2024-03-26T05:15:57.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:57.296+0000 7f249a563700 1 -- 172.21.15.136:0/2875303982 wait complete. 2024-03-26T05:15:57.822 INFO:teuthology.orchestra.run.smithi136.stdout:Generating new minimal ceph.conf... 2024-03-26T05:15:58.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.245+0000 7f42b76a1700 1 Processor -- start 2024-03-26T05:15:58.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.245+0000 7f42b76a1700 1 -- start start 2024-03-26T05:15:58.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.245+0000 7f42b76a1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b0079b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:15:58.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.245+0000 7f42b76a1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b007a140 con 0x7f42b007b710 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.246+0000 7f42b543d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b0079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.246+0000 7f42b543d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b0079b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46282/0 (socket says 172.21.15.136:46282) 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.246+0000 7f42b543d700 1 -- 172.21.15.136:0/3351819610 learned_addr learned my addr 172.21.15.136:0/3351819610 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.246+0000 7f42b543d700 1 -- 172.21.15.136:0/3351819610 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b007a280 con 0x7f42b007b710 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.246+0000 7f42b543d700 1 --2- 172.21.15.136:0/3351819610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b0079b70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f42a8009e80 tx=0x7f42a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f60d9a00191b24 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.246+0000 7f42a7fff700 1 -- 172.21.15.136:0/3351819610 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42a800e070 con 0x7f42b007b710 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.246+0000 7f42a7fff700 1 -- 172.21.15.136:0/3351819610 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f42a800b8c0 con 0x7f42b007b710 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.246+0000 7f42a7fff700 1 -- 172.21.15.136:0/3351819610 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42a80043e0 con 0x7f42b007b710 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.247+0000 7f42b76a1700 1 -- 172.21.15.136:0/3351819610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 msgr2=0x7f42b0079b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.247+0000 7f42b76a1700 1 --2- 172.21.15.136:0/3351819610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b0079b70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f42a8009e80 tx=0x7f42a800b330 comp rx=0 tx=0).stop 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.247+0000 7f42b76a1700 1 -- 172.21.15.136:0/3351819610 shutdown_connections 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.247+0000 7f42b76a1700 1 --2- 172.21.15.136:0/3351819610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b0079b70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:15:58.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.247+0000 7f42b76a1700 1 -- 172.21.15.136:0/3351819610 >> 172.21.15.136:0/3351819610 conn(0x7f42b01016a0 msgr2=0x7f42b0103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.247+0000 7f42b76a1700 1 -- 172.21.15.136:0/3351819610 shutdown_connections 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.247+0000 7f42b76a1700 1 -- 172.21.15.136:0/3351819610 wait complete. 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.248+0000 7f42b76a1700 1 Processor -- start 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.248+0000 7f42b76a1700 1 -- start start 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.248+0000 7f42b76a1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b019c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.248+0000 7f42b76a1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b019c560 con 0x7f42b007b710 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.248+0000 7f42b543d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b019c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.248+0000 7f42b543d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b019c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46298/0 (socket says 172.21.15.136:46298) 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.248+0000 7f42b543d700 1 -- 172.21.15.136:0/3860375435 learned_addr learned my addr 172.21.15.136:0/3860375435 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42b543d700 1 -- 172.21.15.136:0/3860375435 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a8009b30 con 0x7f42b007b710 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42b543d700 1 --2- 172.21.15.136:0/3860375435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b019c020 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f42a8004970 tx=0x7f42a800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42a67fc700 1 -- 172.21.15.136:0/3860375435 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42a800e040 con 0x7f42b007b710 2024-03-26T05:15:58.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42a67fc700 1 -- 172.21.15.136:0/3860375435 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f42a8003e60 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b019c760 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42a67fc700 1 -- 172.21.15.136:0/3860375435 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42a8012790 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b019cc60 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42a67fc700 1 -- 172.21.15.136:0/3860375435 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f42a8025400 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.249+0000 7f42a67fc700 1 -- 172.21.15.136:0/3860375435 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f42a801b7b0 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.250+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b00636b0 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.252+0000 7f42a67fc700 1 -- 172.21.15.136:0/3860375435 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f42a8020020 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.330+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f42b010c4f0 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.330+0000 7f42a67fc700 1 -- 172.21.15.136:0/3860375435 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f42a8017070 con 0x7f42b007b710 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.331+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 msgr2=0x7f42b019c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.331+0000 7f42b76a1700 1 --2- 172.21.15.136:0/3860375435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b019c020 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f42a8004970 tx=0x7f42a800bda0 comp rx=0 tx=0).stop 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.331+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 shutdown_connections 2024-03-26T05:15:58.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.331+0000 7f42b76a1700 1 --2- 172.21.15.136:0/3860375435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42b007b710 0x7f42b019c020 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:15:58.884 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.331+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 >> 172.21.15.136:0/3860375435 conn(0x7f42b01016a0 msgr2=0x7f42b0102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:15:58.884 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.331+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 shutdown_connections 2024-03-26T05:15:58.884 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:15:58.331+0000 7f42b76a1700 1 -- 172.21.15.136:0/3860375435 wait complete. 2024-03-26T05:15:58.884 INFO:teuthology.orchestra.run.smithi136.stdout:Restarting the monitor... 2024-03-26T05:15:58.893 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:58 smithi136 ceph-mon[29174]: from='client.? 172.21.15.136:0/3860375435' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:15:59.174 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:58 smithi136 systemd[1]: Stopping Ceph mon.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:15:59.174 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:59 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a[29170]: 2024-03-26T05:15:59.152+0000 7fb4a7d72700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:15:59.174 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:59 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a[29170]: 2024-03-26T05:15:59.152+0000 7fb4a7d72700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T05:15:59.174 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:59 smithi136 podman[29430]: 2024-03-26 05:15:59.183666521 +0000 UTC m=+0.141378564 container died e2efb7bdd110132eb2c266303fa2c1d0c4379d94334a7878da4a070c7882b60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T05:15:59.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:59 smithi136 podman[29430]: 2024-03-26 05:15:59.544187812 +0000 UTC m=+0.501899852 container cleanup e2efb7bdd110132eb2c266303fa2c1d0c4379d94334a7878da4a070c7882b60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T05:15:59.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:15:59 smithi136 bash[29430]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a 2024-03-26T05:16:00.567 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:00 smithi136 podman[29444]: 2024-03-26 05:16:00.235149136 +0000 UTC m=+1.050011939 container remove e2efb7bdd110132eb2c266303fa2c1d0c4379d94334a7878da4a070c7882b60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:16:00.568 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:00 smithi136 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T05:16:00.568 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:00 smithi136 systemd[1]: Stopped Ceph mon.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:16:00.568 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:00 smithi136 systemd[1]: Starting Ceph mon.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:16:00.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:00 smithi136 podman[29541]: 2024-03-26 05:16:00.480755359 +0000 UTC m=+0.018005121 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T05:16:02.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:01 smithi136 podman[29541]: 2024-03-26 05:16:01.87035462 +0000 UTC m=+1.407604441 container create 8d7627f7b5d6be93f354b15bde43f4d6eec3e4aad15622b6c78742ae60c81319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T05:16:04.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 podman[29541]: 2024-03-26 05:16:03.663950043 +0000 UTC m=+3.201199823 container init 8d7627f7b5d6be93f354b15bde43f4d6eec3e4aad15622b6c78742ae60c81319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T05:16:04.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 podman[29541]: 2024-03-26 05:16:03.668355378 +0000 UTC m=+3.205605140 container start 8d7627f7b5d6be93f354b15bde43f4d6eec3e4aad15622b6c78742ae60c81319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T05:16:04.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T05:16:04.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T05:16:04.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: pidfile_write: ignore empty --pid-file 2024-03-26T05:16:04.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: load: jerasure load: lrc 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: RocksDB version: 7.9.2 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Git sha 0 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: DB SUMMARY 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: DB Session ID: M3LX6ZATDIF6SIQKN2GM 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: CURRENT file: CURRENT 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 87731 ; 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.error_if_exists: 0 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.create_if_missing: 0 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.paranoid_checks: 1 2024-03-26T05:16:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T05:16:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T05:16:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T05:16:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.env: 0x557e4b2ffae0 2024-03-26T05:16:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T05:16:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.info_log: 0x557e4c85b920 2024-03-26T05:16:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T05:16:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.statistics: (nil) 2024-03-26T05:16:04.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.use_fsync: 0 2024-03-26T05:16:04.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_log_file_size: 0 2024-03-26T05:16:04.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T05:16:04.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T05:16:04.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.allow_fallocate: 1 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.use_direct_reads: 0 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.db_log_dir: 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.wal_dir: 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T05:16:04.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.write_buffer_manager: 0x557e4c82e1e0 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.unordered_write: 0 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T05:16:04.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.row_cache: None 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.wal_filter: None 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.two_write_queues: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.wal_compression: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.atomic_flush: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.log_readahead_size: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T05:16:04.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_background_jobs: 2 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_background_compactions: -1 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_subcompactions: 1 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_open_files: -1 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T05:16:04.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_background_flushes: -1 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Compression algorithms supported: 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: kZSTD supported: 0 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: kXpressCompression supported: 0 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: kZlibCompression supported: 1 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: kSnappyCompression supported: 1 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: kLZ4Compression supported: 1 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: kBZip2Compression supported: 0 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T05:16:04.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.merge_operator: 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_filter: None 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557e4c85ba20) 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T05:16:04.083 INFO:journalctl@ceph.mon.a.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_type: 0 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_block_index_type: 0 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_shortening: 1 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: checksum: 4 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: no_block_cache: 0 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache: 0x557e4c85f090 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_options: 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: capacity : 536870912 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: num_shard_bits : 4 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: strict_capacity_limit : 0 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-03-26T05:16:04.084 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_compressed: (nil) 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: persistent_cache: (nil) 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_size: 4096 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_size_deviation: 10 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_restart_interval: 16 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_block_restart_interval: 1 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: metadata_block_size: 4096 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: partition_filters: 0 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: use_delta_encoding: 1 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: filter_policy: bloomfilter 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: whole_key_filtering: 1 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: verify_compression: 0 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: format_version: 5 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: enable_index_compression: 1 2024-03-26T05:16:04.085 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_align: 0 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout: max_auto_readahead_size: 262144 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout: prepopulate_block_cache: 0 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout: initial_auto_readahead_size: 8192 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression: NoCompression 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.num_levels: 7 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T05:16:04.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T05:16:04.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T05:16:04.088 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T05:16:04.089 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.table_properties_collectors: 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.inplace_update_support: 0 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.bloom_locality: 0 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.max_successive_merges: 0 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T05:16:04.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.ttl: 2592000 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.enable_blob_files: false 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.min_blob_size: 0 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T05:16:04.091 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0ded2f6d-98b1-4d6d-aee8-30f2c947183b 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711430163700708, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711430163701645, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 83979, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 266, "table_properties": {"data_size": 82158, "index_size": 231, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10814, "raw_average_key_size": 48, "raw_value_size": 76135, "raw_average_value_size": 338, "num_data_blocks": 10, "num_entries": 225, "num_filter_entries": 225, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711430163, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0ded2f6d-98b1-4d6d-aee8-30f2c947183b", "db_session_id": "M3LX6ZATDIF6SIQKN2GM", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711430163701740, "job": 1, "event": "recovery_finished"} 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557e4c950000 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: DB pointer 0x557e4c93a000 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** DB Stats ** 2024-03-26T05:16:04.092 INFO:journalctl@ceph.mon.a.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** Compaction Stats [default] ** 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: L0 2/0 83.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Sum 2/0 83.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T05:16:04.093 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** Compaction Stats [default] ** 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative compaction: 0.00 GB write, 25.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval compaction: 0.00 GB write, 25.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T05:16:04.094 INFO:journalctl@ceph.mon.a.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: Block cache BinnedLRUCache@0x557e4c85f090#2 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: starting mon.a rank 0 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a@-1(???) e1 preinit fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a@-1(???).mds e1 new map 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a@-1(???).mds e1 print_map 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: e1 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: legacy client fscid: -1 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout: No filesystems configured 2024-03-26T05:16:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: monmap e1: 1 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: fsmap 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T05:16:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:03 smithi136 ceph-mon[29560]: mgrmap e1: no daemons active 2024-03-26T05:16:04.600 INFO:teuthology.orchestra.run.smithi136.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T05:16:05.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:04 smithi136 bash[29541]: 8d7627f7b5d6be93f354b15bde43f4d6eec3e4aad15622b6c78742ae60c81319 2024-03-26T05:16:05.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:04 smithi136 systemd[1]: Started Ceph mon.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:16:09.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1220937645' entity='client.admin' 2024-03-26T05:16:09.201 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.846+0000 7f7214517700 1 Processor -- start 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.846+0000 7f7214517700 1 -- start start 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.846+0000 7f7214517700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c108fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.846+0000 7f7214517700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c079f50 con 0x7f720c106bf0 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.847+0000 7f72122b3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c108fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.847+0000 7f72122b3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c108fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43576/0 (socket says 172.21.15.136:43576) 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.847+0000 7f72122b3700 1 -- 172.21.15.136:0/3613831230 learned_addr learned my addr 172.21.15.136:0/3613831230 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.847+0000 7f72122b3700 1 -- 172.21.15.136:0/3613831230 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c109520 con 0x7f720c106bf0 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.847+0000 7f72122b3700 1 --2- 172.21.15.136:0/3613831230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c108fe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72080098a0 tx=0x7f7208009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d1cbc1ddea9d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.847+0000 7f72112b1700 1 -- 172.21.15.136:0/3613831230 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7208004160 con 0x7f720c106bf0 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.847+0000 7f72112b1700 1 -- 172.21.15.136:0/3613831230 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f72080042c0 con 0x7f720c106bf0 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.847+0000 7f72112b1700 1 -- 172.21.15.136:0/3613831230 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72080119b0 con 0x7f720c106bf0 2024-03-26T05:16:09.202 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.848+0000 7f7214517700 1 -- 172.21.15.136:0/3613831230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 msgr2=0x7f720c108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.848+0000 7f7214517700 1 --2- 172.21.15.136:0/3613831230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c108fe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72080098a0 tx=0x7f7208009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.848+0000 7f7214517700 1 -- 172.21.15.136:0/3613831230 shutdown_connections 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.848+0000 7f7214517700 1 --2- 172.21.15.136:0/3613831230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c108fe0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.848+0000 7f7214517700 1 -- 172.21.15.136:0/3613831230 >> 172.21.15.136:0/3613831230 conn(0x7f720c1005c0 msgr2=0x7f720c102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.848+0000 7f7214517700 1 -- 172.21.15.136:0/3613831230 shutdown_connections 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.848+0000 7f7214517700 1 -- 172.21.15.136:0/3613831230 wait complete. 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.848+0000 7f7214517700 1 Processor -- start 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f7214517700 1 -- start start 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f7214517700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f7214517700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c19a380 con 0x7f720c106bf0 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f72122b3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f72122b3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43582/0 (socket says 172.21.15.136:43582) 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f72122b3700 1 -- 172.21.15.136:0/1220937645 learned_addr learned my addr 172.21.15.136:0/1220937645 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:09.203 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f72122b3700 1 -- 172.21.15.136:0/1220937645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7208009550 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f72122b3700 1 --2- 172.21.15.136:0/1220937645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c199e40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7208009130 tx=0x7f7208011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f72037fe700 1 -- 172.21.15.136:0/1220937645 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7208003aa0 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f72037fe700 1 -- 172.21.15.136:0/1220937645 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f720801a8c0 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f72037fe700 1 -- 172.21.15.136:0/1220937645 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7208019790 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720c19a580 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.849+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720c19aa20 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.850+0000 7f72037fe700 1 -- 172.21.15.136:0/1220937645 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7208025460 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.850+0000 7f72037fe700 1 -- 172.21.15.136:0/1220937645 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f720800bb20 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.850+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720c10a2c0 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.853+0000 7f72037fe700 1 -- 172.21.15.136:0/1220937645 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f720802a060 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.919+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f720c19d360 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.921+0000 7f72037fe700 1 -- 172.21.15.136:0/1220937645 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f720801aa30 con 0x7f720c106bf0 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.922+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 msgr2=0x7f720c199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:09.204 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.922+0000 7f7214517700 1 --2- 172.21.15.136:0/1220937645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c199e40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7208009130 tx=0x7f7208011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:09.205 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.922+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 shutdown_connections 2024-03-26T05:16:09.205 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.922+0000 7f7214517700 1 --2- 172.21.15.136:0/1220937645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f720c106bf0 0x7f720c199e40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:09.205 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.922+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 >> 172.21.15.136:0/1220937645 conn(0x7f720c1005c0 msgr2=0x7f720c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:09.205 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.922+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 shutdown_connections 2024-03-26T05:16:09.205 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:07.922+0000 7f7214517700 1 -- 172.21.15.136:0/1220937645 wait complete. 2024-03-26T05:16:09.205 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T05:16:09.218 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:16:09.218 INFO:teuthology.orchestra.run.smithi136.stdout:Creating mgr... 2024-03-26T05:16:09.219 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T05:16:09.220 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T05:16:09.466 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.y 2024-03-26T05:16:09.466 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T05:16:09.610 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715.target.wants/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@.service. 2024-03-26T05:16:09.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:09 smithi136 systemd[1]: Starting Ceph mgr.y for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:16:10.132 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:09 smithi136 podman[29778]: 2024-03-26 05:16:09.756625561 +0000 UTC m=+0.017883734 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T05:16:10.132 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:09 smithi136 podman[29778]: 2024-03-26 05:16:09.856391385 +0000 UTC m=+0.117649559 container create 00c92f6e2086c6feb56e24f2403693939ac920c0e68af51d9cbb66b43f774cbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T05:16:10.132 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:10 smithi136 podman[29778]: 2024-03-26 05:16:10.058518902 +0000 UTC m=+0.319777075 container init 00c92f6e2086c6feb56e24f2403693939ac920c0e68af51d9cbb66b43f774cbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T05:16:10.132 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:10 smithi136 podman[29778]: 2024-03-26 05:16:10.060913569 +0000 UTC m=+0.322171743 container start 00c92f6e2086c6feb56e24f2403693939ac920c0e68af51d9cbb66b43f774cbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-03-26T05:16:10.132 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:10 smithi136 bash[29778]: 00c92f6e2086c6feb56e24f2403693939ac920c0e68af51d9cbb66b43f774cbc 2024-03-26T05:16:10.144 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T05:16:10.144 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-03-26T05:16:10.151 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T05:16:10.152 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T05:16:10.152 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-03-26T05:16:10.152 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T05:16:10.159 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T05:16:10.160 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-03-26T05:16:10.167 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T05:16:10.167 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T05:16:10.167 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-03-26T05:16:10.167 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T05:16:10.167 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr to start... 2024-03-26T05:16:10.167 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr... 2024-03-26T05:16:10.418 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:10 smithi136 systemd[1]: Started Ceph mgr.y for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:16:10.678 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:10 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:10.612+0000 7f5f3a380200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T05:16:10.987 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/483170632' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T05:16:10.987 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:10 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:10.709+0000 7f5f3a380200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T05:16:11.265 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:16:11.267 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T05:16:11.267 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "d69b39a8-eb2f-11ee-95d0-87774f69a715", 2024-03-26T05:16:11.267 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T05:16:11.267 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T05:16:11.268 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:11.269 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T05:16:11.269 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:11.269 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T05:16:11.269 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T05:16:11.269 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T05:16:11.276 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T05:16:11.276 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T05:16:11.276 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T05:16:11.276 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:11.276 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T05:16:11.276 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T05:16:11.276 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T05:16:11.277 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T05:15:55.713726+0000", 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:11.278 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.609+0000 7f2e31d5d700 1 Processor -- start 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.610+0000 7f2e31d5d700 1 -- start start 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.610+0000 7f2e31d5d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.610+0000 7f2e31d5d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c1059f0 con 0x7f2e2c105000 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.610+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.610+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43596/0 (socket says 172.21.15.136:43596) 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.610+0000 7f2e2b7fe700 1 -- 172.21.15.136:0/1622661818 learned_addr learned my addr 172.21.15.136:0/1622661818 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.610+0000 7f2e2b7fe700 1 -- 172.21.15.136:0/1622661818 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c106200 con 0x7f2e2c105000 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.611+0000 7f2e2b7fe700 1 --2- 172.21.15.136:0/1622661818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c105420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0098a0 tx=0x7f2e1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d57045c028554f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.611+0000 7f2e2a7fc700 1 -- 172.21.15.136:0/1622661818 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1c004160 con 0x7f2e2c105000 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.611+0000 7f2e2a7fc700 1 -- 172.21.15.136:0/1622661818 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f2e1c0042c0 con 0x7f2e2c105000 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.611+0000 7f2e2a7fc700 1 -- 172.21.15.136:0/1622661818 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1c0119b0 con 0x7f2e2c105000 2024-03-26T05:16:11.279 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.612+0000 7f2e31d5d700 1 -- 172.21.15.136:0/1622661818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 msgr2=0x7f2e2c105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.612+0000 7f2e31d5d700 1 --2- 172.21.15.136:0/1622661818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c105420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0098a0 tx=0x7f2e1c009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.612+0000 7f2e31d5d700 1 -- 172.21.15.136:0/1622661818 shutdown_connections 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.612+0000 7f2e31d5d700 1 --2- 172.21.15.136:0/1622661818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c105420 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.612+0000 7f2e31d5d700 1 -- 172.21.15.136:0/1622661818 >> 172.21.15.136:0/1622661818 conn(0x7f2e2c100580 msgr2=0x7f2e2c1029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.612+0000 7f2e31d5d700 1 -- 172.21.15.136:0/1622661818 shutdown_connections 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.612+0000 7f2e31d5d700 1 -- 172.21.15.136:0/1622661818 wait complete. 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.613+0000 7f2e31d5d700 1 Processor -- start 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.613+0000 7f2e31d5d700 1 -- start start 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.613+0000 7f2e31d5d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c079b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.613+0000 7f2e31d5d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c07a090 con 0x7f2e2c105000 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.613+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c079b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.613+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c079b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43608/0 (socket says 172.21.15.136:43608) 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.613+0000 7f2e2b7fe700 1 -- 172.21.15.136:0/483170632 learned_addr learned my addr 172.21.15.136:0/483170632 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e2b7fe700 1 -- 172.21.15.136:0/483170632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c009550 con 0x7f2e2c105000 2024-03-26T05:16:11.280 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e2b7fe700 1 --2- 172.21.15.136:0/483170632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c079b50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c006af0 tx=0x7f2e1c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e28ff9700 1 -- 172.21.15.136:0/483170632 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1c0039e0 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c19cbb0 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e28ff9700 1 -- 172.21.15.136:0/483170632 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f2e1c003b40 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e28ff9700 1 -- 172.21.15.136:0/483170632 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1c019720 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c19d050 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e28ff9700 1 -- 172.21.15.136:0/483170632 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2e1c024410 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.614+0000 7f2e28ff9700 1 -- 172.21.15.136:0/483170632 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2e1c00b8b0 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.615+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c0636b0 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.617+0000 7f2e28ff9700 1 -- 172.21.15.136:0/483170632 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2e1c019d60 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.688+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2e2c109550 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.688+0000 7f2e28ff9700 1 -- 172.21.15.136:0/483170632 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f2e1c01aba0 con 0x7f2e2c105000 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.689+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 msgr2=0x7f2e2c079b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.689+0000 7f2e31d5d700 1 --2- 172.21.15.136:0/483170632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c079b50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c006af0 tx=0x7f2e1c011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:11.281 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.689+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 shutdown_connections 2024-03-26T05:16:11.282 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.689+0000 7f2e31d5d700 1 --2- 172.21.15.136:0/483170632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e2c105000 0x7f2e2c079b50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:11.282 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.689+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 >> 172.21.15.136:0/483170632 conn(0x7f2e2c100580 msgr2=0x7f2e2c1010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:11.282 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.690+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 shutdown_connections 2024-03-26T05:16:11.282 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:10.690+0000 7f2e31d5d700 1 -- 172.21.15.136:0/483170632 wait complete. 2024-03-26T05:16:11.282 INFO:teuthology.orchestra.run.smithi136.stdout:mgr not available, waiting (1/15)... 2024-03-26T05:16:11.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:11 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:11.354+0000 7f5f3a380200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T05:16:11.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:11 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:11.445+0000 7f5f3a380200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T05:16:12.008 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:11 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:11.584+0000 7f5f3a380200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T05:16:12.008 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:11 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:11.677+0000 7f5f3a380200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T05:16:12.008 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:11 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:11.759+0000 7f5f3a380200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T05:16:12.009 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:12 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:12.018+0000 7f5f3a380200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T05:16:12.291 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:12 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:12.125+0000 7f5f3a380200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T05:16:12.291 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:12 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:12.208+0000 7f5f3a380200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T05:16:12.291 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:12 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:12.300+0000 7f5f3a380200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T05:16:12.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:12 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:12.474+0000 7f5f3a380200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T05:16:13.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:13 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:13.429+0000 7f5f3a380200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T05:16:13.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:13 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:13.561+0000 7f5f3a380200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T05:16:14.163 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:13 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:13.895+0000 7f5f3a380200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T05:16:14.163 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:13 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:13.985+0000 7f5f3a380200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T05:16:14.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/483987836' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T05:16:14.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:14 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:14.187+0000 7f5f3a380200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:16:14.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:14 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:14.366+0000 7f5f3a380200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T05:16:14.894 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:14 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:14.645+0000 7f5f3a380200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:16:14.895 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:14 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:14.904+0000 7f5f3a380200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T05:16:15.324 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:15 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:15.137+0000 7f5f3a380200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T05:16:16.323 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:15 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:15.926+0000 7f5f3a380200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T05:16:16.767 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:16.489+0000 7f5f3a380200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T05:16:16.767 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:16.584+0000 7f5f3a380200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T05:16:17.043 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:16.805+0000 7f5f3a380200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T05:16:17.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: Activating manager daemon y 2024-03-26T05:16:17.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: mgrmap e2: y(active, starting, since 0.00311479s) 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: Manager daemon y is now available 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' 2024-03-26T05:16:17.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:16 smithi136 ceph-mon[29560]: from='mgr.14100 172.21.15.136:0/2000238269' entity='mgr.y' 2024-03-26T05:16:17.290 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:16:17.290 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T05:16:17.290 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "d69b39a8-eb2f-11ee-95d0-87774f69a715", 2024-03-26T05:16:17.290 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T05:16:17.290 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T05:16:17.290 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T05:16:17.290 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:17.291 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T05:16:17.292 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:17.292 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T05:16:17.292 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T05:16:17.292 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T05:16:17.299 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T05:16:17.299 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T05:16:17.299 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T05:16:17.299 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T05:16:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T05:15:55.713726+0000", 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T05:16:17.301 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.040+0000 7f1980d50700 1 Processor -- start 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.040+0000 7f1980d50700 1 -- start start 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.040+0000 7f1980d50700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c108f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.040+0000 7f1980d50700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c079720 con 0x7f197c106b40 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.040+0000 7f197a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c108f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.040+0000 7f197a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c108f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37328/0 (socket says 172.21.15.136:37328) 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.040+0000 7f197a59c700 1 -- 172.21.15.136:0/3942527796 learned_addr learned my addr 172.21.15.136:0/3942527796 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.041+0000 7f197a59c700 1 -- 172.21.15.136:0/3942527796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c109470 con 0x7f197c106b40 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.041+0000 7f197a59c700 1 --2- 172.21.15.136:0/3942527796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c108f30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f196c0098a0 tx=0x7f196c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4b13320287dace server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.041+0000 7f197959a700 1 -- 172.21.15.136:0/3942527796 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f196c004160 con 0x7f197c106b40 2024-03-26T05:16:17.302 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.041+0000 7f197959a700 1 -- 172.21.15.136:0/3942527796 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f196c0042c0 con 0x7f197c106b40 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.041+0000 7f197959a700 1 -- 172.21.15.136:0/3942527796 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f196c0119b0 con 0x7f197c106b40 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.041+0000 7f1980d50700 1 -- 172.21.15.136:0/3942527796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 msgr2=0x7f197c108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.041+0000 7f1980d50700 1 --2- 172.21.15.136:0/3942527796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c108f30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f196c0098a0 tx=0x7f196c009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 -- 172.21.15.136:0/3942527796 shutdown_connections 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 --2- 172.21.15.136:0/3942527796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c108f30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 -- 172.21.15.136:0/3942527796 >> 172.21.15.136:0/3942527796 conn(0x7f197c100590 msgr2=0x7f197c1029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 -- 172.21.15.136:0/3942527796 shutdown_connections 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 -- 172.21.15.136:0/3942527796 wait complete. 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 Processor -- start 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 -- start start 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c1063c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.042+0000 7f1980d50700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c106900 con 0x7f197c106b40 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f197a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c1063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f197a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c1063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37338/0 (socket says 172.21.15.136:37338) 2024-03-26T05:16:17.303 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f197a59c700 1 -- 172.21.15.136:0/483987836 learned_addr learned my addr 172.21.15.136:0/483987836 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f197a59c700 1 -- 172.21.15.136:0/483987836 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196c009550 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f197a59c700 1 --2- 172.21.15.136:0/483987836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c1063c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f196c0046c0 tx=0x7f196c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f196b7fe700 1 -- 172.21.15.136:0/483987836 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f196c0039c0 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f196b7fe700 1 -- 172.21.15.136:0/483987836 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f196c003b20 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c104af0 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f196b7fe700 1 -- 172.21.15.136:0/483987836 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f196c01a7b0 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c104f90 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f196b7fe700 1 -- 172.21.15.136:0/483987836 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f196c01aa20 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.043+0000 7f196b7fe700 1 -- 172.21.15.136:0/483987836 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f196c019700 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.044+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c0636b0 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.048+0000 7f196b7fe700 1 -- 172.21.15.136:0/483987836 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f196c02de40 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.121+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f197c10a4a0 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.121+0000 7f196b7fe700 1 -- 172.21.15.136:0/483987836 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f196c024b30 con 0x7f197c106b40 2024-03-26T05:16:17.304 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.122+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 msgr2=0x7f197c1063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:17.305 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.122+0000 7f1980d50700 1 --2- 172.21.15.136:0/483987836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c1063c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f196c0046c0 tx=0x7f196c011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:17.305 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.122+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 shutdown_connections 2024-03-26T05:16:17.305 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.122+0000 7f1980d50700 1 --2- 172.21.15.136:0/483987836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f197c106b40 0x7f197c1063c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:17.305 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.122+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 >> 172.21.15.136:0/483987836 conn(0x7f197c100590 msgr2=0x7f197c101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:17.305 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.122+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 shutdown_connections 2024-03-26T05:16:17.305 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:14.122+0000 7f1980d50700 1 -- 172.21.15.136:0/483987836 wait complete. 2024-03-26T05:16:17.305 INFO:teuthology.orchestra.run.smithi136.stdout:mgr not available, waiting (2/15)... 2024-03-26T05:16:19.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:18 smithi136 ceph-mon[29560]: mgrmap e3: y(active, since 1.00671s) 2024-03-26T05:16:20.579 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:16:20.579 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T05:16:20.579 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "d69b39a8-eb2f-11ee-95d0-87774f69a715", 2024-03-26T05:16:20.579 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T05:16:20.579 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T05:16:20.579 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T05:16:20.580 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T05:16:20.581 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T05:16:20.582 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T05:15:55.713726+0000", 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T05:16:20.583 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.730+0000 7f7cbb26a700 1 Processor -- start 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.730+0000 7f7cbb26a700 1 -- start start 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.730+0000 7f7cbb26a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb4104e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.730+0000 7f7cbb26a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4079f50 con 0x7f7cb41049e0 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.730+0000 7f7cb9006700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb4104e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.730+0000 7f7cb9006700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb4104e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37414/0 (socket says 172.21.15.136:37414) 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.730+0000 7f7cb9006700 1 -- 172.21.15.136:0/2385128851 learned_addr learned my addr 172.21.15.136:0/2385128851 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.730+0000 7f7cb9006700 1 -- 172.21.15.136:0/2385128851 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb4105340 con 0x7f7cb41049e0 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cb9006700 1 --2- 172.21.15.136:0/2385128851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb4104e00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00098a0 tx=0x7f7cb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1da104d5f63d70f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cabfff700 1 -- 172.21.15.136:0/2385128851 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7cb0004160 con 0x7f7cb41049e0 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cabfff700 1 -- 172.21.15.136:0/2385128851 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f7cb00042c0 con 0x7f7cb41049e0 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cabfff700 1 -- 172.21.15.136:0/2385128851 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7cb00119b0 con 0x7f7cb41049e0 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2385128851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 msgr2=0x7f7cb4104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cbb26a700 1 --2- 172.21.15.136:0/2385128851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb4104e00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00098a0 tx=0x7f7cb0009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2385128851 shutdown_connections 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cbb26a700 1 --2- 172.21.15.136:0/2385128851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb4104e00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2385128851 >> 172.21.15.136:0/2385128851 conn(0x7f7cb41005e0 msgr2=0x7f7cb4102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2385128851 shutdown_connections 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.731+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2385128851 wait complete. 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.732+0000 7f7cbb26a700 1 Processor -- start 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.732+0000 7f7cbb26a700 1 -- start start 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.732+0000 7f7cbb26a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb419c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.732+0000 7f7cbb26a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb419c580 con 0x7f7cb41049e0 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.732+0000 7f7cb9006700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb419c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.732+0000 7f7cb9006700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb419c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37428/0 (socket says 172.21.15.136:37428) 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.732+0000 7f7cb9006700 1 -- 172.21.15.136:0/2964539288 learned_addr learned my addr 172.21.15.136:0/2964539288 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.733+0000 7f7cb9006700 1 -- 172.21.15.136:0/2964539288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb0009550 con 0x7f7cb41049e0 2024-03-26T05:16:20.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.733+0000 7f7cb9006700 1 --2- 172.21.15.136:0/2964539288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb419c040 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00046c0 tx=0x7f7cb0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.733+0000 7f7caa7fc700 1 -- 172.21.15.136:0/2964539288 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7cb00039c0 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.733+0000 7f7caa7fc700 1 -- 172.21.15.136:0/2964539288 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f7cb0003b20 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.733+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb419c780 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.733+0000 7f7caa7fc700 1 -- 172.21.15.136:0/2964539288 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7cb001a7b0 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.733+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb419cc20 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.734+0000 7f7caa7fc700 1 -- 172.21.15.136:0/2964539288 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 3) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f7cb001a910 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.734+0000 7f7caa7fc700 1 --2- 172.21.15.136:0/2964539288 >> v2:172.21.15.136:6800/1232474076 conn(0x7f7ca0038d40 0x7f7ca003b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.734+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb4195d10 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.734+0000 7f7caa7fc700 1 -- 172.21.15.136:0/2964539288 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7cb004cdc0 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.734+0000 7f7cb8805700 1 --2- 172.21.15.136:0/2964539288 >> v2:172.21.15.136:6800/1232474076 conn(0x7f7ca0038d40 0x7f7ca003b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.734+0000 7f7cb8805700 1 --2- 172.21.15.136:0/2964539288 >> v2:172.21.15.136:6800/1232474076 conn(0x7f7ca0038d40 0x7f7ca003b200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4006fd0 tx=0x7f7ca4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.740+0000 7f7caa7fc700 1 -- 172.21.15.136:0/2964539288 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7cb0023390 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:19.823+0000 7f7caa7fc700 1 -- 172.21.15.136:0/2964539288 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mgrmap(e 4) v1 ==== 45882+0+0 (secure 0 0 0) 0x7f7cb002d910 con 0x7f7cb41049e0 2024-03-26T05:16:20.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.009+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7cb41061b0 con 0x7f7cb41049e0 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.010+0000 7f7caa7fc700 1 -- 172.21.15.136:0/2964539288 <== mon.0 v2:172.21.15.136:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f7cb0018dc0 con 0x7f7cb41049e0 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 >> v2:172.21.15.136:6800/1232474076 conn(0x7f7ca0038d40 msgr2=0x7f7ca003b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 --2- 172.21.15.136:0/2964539288 >> v2:172.21.15.136:6800/1232474076 conn(0x7f7ca0038d40 0x7f7ca003b200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4006fd0 tx=0x7f7ca4006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 msgr2=0x7f7cb419c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 --2- 172.21.15.136:0/2964539288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb419c040 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00046c0 tx=0x7f7cb0011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 shutdown_connections 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 --2- 172.21.15.136:0/2964539288 >> v2:172.21.15.136:6800/1232474076 conn(0x7f7ca0038d40 0x7f7ca003b200 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 --2- 172.21.15.136:0/2964539288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cb41049e0 0x7f7cb419c040 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 >> 172.21.15.136:0/2964539288 conn(0x7f7cb41005e0 msgr2=0x7f7cb4192e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 shutdown_connections 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.013+0000 7f7cbb26a700 1 -- 172.21.15.136:0/2964539288 wait complete. 2024-03-26T05:16:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:mgr is available 2024-03-26T05:16:20.985 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:20 smithi136 ceph-mon[29560]: mgrmap e4: y(active, since 3s) 2024-03-26T05:16:20.985 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2964539288' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T05:16:21.821 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout fsid = d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T05:16:21.822 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.999+0000 7f6bdde16700 1 Processor -- start 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.999+0000 7f6bdde16700 1 -- start start 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.999+0000 7f6bdde16700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8104b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.999+0000 7f6bdde16700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd8105150 con 0x7f6bd8104760 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:20.999+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8104b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.000+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8104b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36840/0 (socket says 172.21.15.136:36840) 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.000+0000 7f6bd77fe700 1 -- 172.21.15.136:0/3419708094 learned_addr learned my addr 172.21.15.136:0/3419708094 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.000+0000 7f6bd77fe700 1 -- 172.21.15.136:0/3419708094 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd8105960 con 0x7f6bd8104760 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.000+0000 7f6bd77fe700 1 --2- 172.21.15.136:0/3419708094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8104b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800af90 tx=0x7f6bc800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f59cd6104123ba server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.000+0000 7f6bd67fc700 1 -- 172.21.15.136:0/3419708094 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bc800f070 con 0x7f6bd8104760 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.000+0000 7f6bd67fc700 1 -- 172.21.15.136:0/3419708094 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f6bc800c9a0 con 0x7f6bd8104760 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.000+0000 7f6bd67fc700 1 -- 172.21.15.136:0/3419708094 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bc8004460 con 0x7f6bd8104760 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 -- 172.21.15.136:0/3419708094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 msgr2=0x7f6bd8104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 --2- 172.21.15.136:0/3419708094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8104b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800af90 tx=0x7f6bc800c330 comp rx=0 tx=0).stop 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 -- 172.21.15.136:0/3419708094 shutdown_connections 2024-03-26T05:16:21.823 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 --2- 172.21.15.136:0/3419708094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8104b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 -- 172.21.15.136:0/3419708094 >> 172.21.15.136:0/3419708094 conn(0x7f6bd80ffd20 msgr2=0x7f6bd8102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 -- 172.21.15.136:0/3419708094 shutdown_connections 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 -- 172.21.15.136:0/3419708094 wait complete. 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 Processor -- start 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.001+0000 7f6bdde16700 1 -- start start 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bdde16700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bdde16700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd81980e0 con 0x7f6bd8104760 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36848/0 (socket says 172.21.15.136:36848) 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bd77fe700 1 -- 172.21.15.136:0/2485522681 learned_addr learned my addr 172.21.15.136:0/2485522681 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bd77fe700 1 -- 172.21.15.136:0/2485522681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc800ac40 con 0x7f6bd8104760 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bd77fe700 1 --2- 172.21.15.136:0/2485522681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8197ba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800af60 tx=0x7f6bc8004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bd4ff9700 1 -- 172.21.15.136:0/2485522681 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bc800f050 con 0x7f6bd8104760 2024-03-26T05:16:21.824 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd81982e0 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bd4ff9700 1 -- 172.21.15.136:0/2485522681 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f6bc8003d60 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bd4ff9700 1 -- 172.21.15.136:0/2485522681 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bc801e8d0 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.002+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd81987e0 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.003+0000 7f6bd4ff9700 1 -- 172.21.15.136:0/2485522681 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 4) v1 ==== 45882+0+0 (secure 0 0 0) 0x7f6bc801a040 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.003+0000 7f6bd4ff9700 1 --2- 172.21.15.136:0/2485522681 >> v2:172.21.15.136:6800/1232474076 conn(0x7f6bc0038e40 0x7f6bc003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.003+0000 7f6bd4ff9700 1 -- 172.21.15.136:0/2485522681 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6bc804dd90 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.003+0000 7f6bd6ffd700 1 --2- 172.21.15.136:0/2485522681 >> v2:172.21.15.136:6800/1232474076 conn(0x7f6bc0038e40 0x7f6bc003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.003+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd8191840 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.004+0000 7f6bd6ffd700 1 --2- 172.21.15.136:0/2485522681 >> v2:172.21.15.136:6800/1232474076 conn(0x7f6bc0038e40 0x7f6bc003b300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc006fd0 tx=0x7f6bcc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.009+0000 7f6bd4ff9700 1 -- 172.21.15.136:0/2485522681 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6bc8018080 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.198+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6bd80636b0 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.199+0000 7f6bd4ff9700 1 -- 172.21.15.136:0/2485522681 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7f6bc8022020 con 0x7f6bd8104760 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 >> v2:172.21.15.136:6800/1232474076 conn(0x7f6bc0038e40 msgr2=0x7f6bc003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:21.825 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 --2- 172.21.15.136:0/2485522681 >> v2:172.21.15.136:6800/1232474076 conn(0x7f6bc0038e40 0x7f6bc003b300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc006fd0 tx=0x7f6bcc006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 msgr2=0x7f6bd8197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 --2- 172.21.15.136:0/2485522681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8197ba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800af60 tx=0x7f6bc8004a00 comp rx=0 tx=0).stop 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 shutdown_connections 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 --2- 172.21.15.136:0/2485522681 >> v2:172.21.15.136:6800/1232474076 conn(0x7f6bc0038e40 0x7f6bc003b300 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 --2- 172.21.15.136:0/2485522681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd8104760 0x7f6bd8197ba0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 >> 172.21.15.136:0/2485522681 conn(0x7f6bd80ffd20 msgr2=0x7f6bd818e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 shutdown_connections 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:21.201+0000 7f6bdde16700 1 -- 172.21.15.136:0/2485522681 wait complete. 2024-03-26T05:16:21.826 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling cephadm module... 2024-03-26T05:16:22.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2485522681' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T05:16:23.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1711823672' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T05:16:23.074 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:22 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ignoring --setuser ceph since I am not root 2024-03-26T05:16:23.074 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:22 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ignoring --setgroup ceph since I am not root 2024-03-26T05:16:23.393 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc162def700 1 Processor -- start 2024-03-26T05:16:23.393 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc162def700 1 -- start start 2024-03-26T05:16:23.393 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc162def700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:23.393 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc162def700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c106ae0 con 0x7fc15c1060f0 2024-03-26T05:16:23.393 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc160b8b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc160b8b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36862/0 (socket says 172.21.15.136:36862) 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc160b8b700 1 -- 172.21.15.136:0/567943099 learned_addr learned my addr 172.21.15.136:0/567943099 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc160b8b700 1 -- 172.21.15.136:0/567943099 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c1072f0 con 0x7fc15c1060f0 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc160b8b700 1 --2- 172.21.15.136:0/567943099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c106510 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc1500098a0 tx=0x7fc150009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d816d9fa2a7cd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc15b7fe700 1 -- 172.21.15.136:0/567943099 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc150004160 con 0x7fc15c1060f0 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc15b7fe700 1 -- 172.21.15.136:0/567943099 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fc1500042c0 con 0x7fc15c1060f0 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.274+0000 7fc15b7fe700 1 -- 172.21.15.136:0/567943099 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1500119b0 con 0x7fc15c1060f0 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.275+0000 7fc162def700 1 -- 172.21.15.136:0/567943099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 msgr2=0x7fc15c106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.275+0000 7fc162def700 1 --2- 172.21.15.136:0/567943099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c106510 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc1500098a0 tx=0x7fc150009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.275+0000 7fc162def700 1 -- 172.21.15.136:0/567943099 shutdown_connections 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.275+0000 7fc162def700 1 --2- 172.21.15.136:0/567943099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c106510 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.275+0000 7fc162def700 1 -- 172.21.15.136:0/567943099 >> 172.21.15.136:0/567943099 conn(0x7fc15c1016b0 msgr2=0x7fc15c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:23.394 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.275+0000 7fc162def700 1 -- 172.21.15.136:0/567943099 shutdown_connections 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.275+0000 7fc162def700 1 -- 172.21.15.136:0/567943099 wait complete. 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc162def700 1 Processor -- start 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc162def700 1 -- start start 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc162def700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c07bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc162def700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c07c1b0 con 0x7fc15c1060f0 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc160b8b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c07bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc160b8b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c07bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36874/0 (socket says 172.21.15.136:36874) 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc160b8b700 1 -- 172.21.15.136:0/1711823672 learned_addr learned my addr 172.21.15.136:0/1711823672 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc160b8b700 1 -- 172.21.15.136:0/1711823672 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc150009550 con 0x7fc15c1060f0 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc160b8b700 1 --2- 172.21.15.136:0/1711823672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c07bc70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc1500046c0 tx=0x7fc150011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc159ffb700 1 -- 172.21.15.136:0/1711823672 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1500039c0 con 0x7fc15c1060f0 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc159ffb700 1 -- 172.21.15.136:0/1711823672 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fc150003b20 con 0x7fc15c1060f0 2024-03-26T05:16:23.395 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc159ffb700 1 -- 172.21.15.136:0/1711823672 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc15001a780 con 0x7fc15c1060f0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.276+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15c07a3a0 con 0x7fc15c1060f0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.277+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15c07a840 con 0x7fc15c1060f0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.277+0000 7fc159ffb700 1 -- 172.21.15.136:0/1711823672 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 4) v1 ==== 45882+0+0 (secure 0 0 0) 0x7fc150024b50 con 0x7fc15c1060f0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.278+0000 7fc159ffb700 1 --2- 172.21.15.136:0/1711823672 >> v2:172.21.15.136:6800/1232474076 conn(0x7fc14c038e50 0x7fc14c03b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.278+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c0636b0 con 0x7fc15c1060f0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.278+0000 7fc159ffb700 1 -- 172.21.15.136:0/1711823672 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc15004cfb0 con 0x7fc15c1060f0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.278+0000 7fc15bfff700 1 --2- 172.21.15.136:0/1711823672 >> v2:172.21.15.136:6800/1232474076 conn(0x7fc14c038e50 0x7fc14c03b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.278+0000 7fc15bfff700 1 --2- 172.21.15.136:0/1711823672 >> v2:172.21.15.136:6800/1232474076 conn(0x7fc14c038e50 0x7fc14c03b310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc148006fd0 tx=0x7fc148006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.283+0000 7fc159ffb700 1 -- 172.21.15.136:0/1711823672 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc15007b280 con 0x7fc15c1060f0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.531+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fc15c07ab40 con 0x7fc15c1060f0 2024-03-26T05:16:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.827+0000 7fc159ffb700 1 -- 172.21.15.136:0/1711823672 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fc150024430 con 0x7fc15c1060f0 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.829+0000 7fc159ffb700 1 -- 172.21.15.136:0/1711823672 <== mon.0 v2:172.21.15.136:3300/0 8 ==== mgrmap(e 5) v1 ==== 45893+0+0 (secure 0 0 0) 0x7fc15002d910 con 0x7fc15c1060f0 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 >> v2:172.21.15.136:6800/1232474076 conn(0x7fc14c038e50 msgr2=0x7fc14c03b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 --2- 172.21.15.136:0/1711823672 >> v2:172.21.15.136:6800/1232474076 conn(0x7fc14c038e50 0x7fc14c03b310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc148006fd0 tx=0x7fc148006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 msgr2=0x7fc15c07bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 --2- 172.21.15.136:0/1711823672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c07bc70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc1500046c0 tx=0x7fc150011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 shutdown_connections 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 --2- 172.21.15.136:0/1711823672 >> v2:172.21.15.136:6800/1232474076 conn(0x7fc14c038e50 0x7fc14c03b310 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 --2- 172.21.15.136:0/1711823672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc15c1060f0 0x7fc15c07bc70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 >> 172.21.15.136:0/1711823672 conn(0x7fc15c1016b0 msgr2=0x7fc15c102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 shutdown_connections 2024-03-26T05:16:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:22.830+0000 7fc162def700 1 -- 172.21.15.136:0/1711823672 wait complete. 2024-03-26T05:16:23.705 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:23 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:23.382+0000 7f0fc6fc9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T05:16:23.705 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:23 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:23.482+0000 7f0fc6fc9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T05:16:24.009 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1711823672' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T05:16:24.009 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:23 smithi136 ceph-mon[29560]: mgrmap e5: y(active, since 6s) 2024-03-26T05:16:24.324 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:24 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:24.150+0000 7f0fc6fc9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T05:16:24.324 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:24 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:24.242+0000 7f0fc6fc9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T05:16:24.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T05:16:24.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T05:16:24.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T05:16:24.584 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.813+0000 7f272c6ac700 1 Processor -- start 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.813+0000 7f272c6ac700 1 -- start start 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.813+0000 7f272c6ac700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272407a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.813+0000 7f272c6ac700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272407a9d0 con 0x7f272407bfa0 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.813+0000 7f272a448700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272407a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.813+0000 7f272a448700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272407a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36892/0 (socket says 172.21.15.136:36892) 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.813+0000 7f272a448700 1 -- 172.21.15.136:0/141929197 learned_addr learned my addr 172.21.15.136:0/141929197 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.813+0000 7f272a448700 1 -- 172.21.15.136:0/141929197 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272407ab10 con 0x7f272407bfa0 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.814+0000 7f272a448700 1 --2- 172.21.15.136:0/141929197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272407a400 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f27200098a0 tx=0x7f2720009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1230becb33b95bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.814+0000 7f2729446700 1 -- 172.21.15.136:0/141929197 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2720004160 con 0x7f272407bfa0 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.814+0000 7f2729446700 1 -- 172.21.15.136:0/141929197 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f27200042c0 con 0x7f272407bfa0 2024-03-26T05:16:24.585 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.814+0000 7f2729446700 1 -- 172.21.15.136:0/141929197 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27200119b0 con 0x7f272407bfa0 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.814+0000 7f272c6ac700 1 -- 172.21.15.136:0/141929197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 msgr2=0x7f272407a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.814+0000 7f272c6ac700 1 --2- 172.21.15.136:0/141929197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272407a400 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f27200098a0 tx=0x7f2720009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 -- 172.21.15.136:0/141929197 shutdown_connections 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 --2- 172.21.15.136:0/141929197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272407a400 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 -- 172.21.15.136:0/141929197 >> 172.21.15.136:0/141929197 conn(0x7f2724101670 msgr2=0x7f2724103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 -- 172.21.15.136:0/141929197 shutdown_connections 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 -- 172.21.15.136:0/141929197 wait complete. 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 Processor -- start 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 -- start start 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.815+0000 7f272c6ac700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272419c600 con 0x7f272407bfa0 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f272a448700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f272a448700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272419c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36906/0 (socket says 172.21.15.136:36906) 2024-03-26T05:16:24.586 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f272a448700 1 -- 172.21.15.136:0/202798501 learned_addr learned my addr 172.21.15.136:0/202798501 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f272a448700 1 -- 172.21.15.136:0/202798501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2720009550 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f272a448700 1 --2- 172.21.15.136:0/202798501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272419c0c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f27200046c0 tx=0x7f2720011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f271b7fe700 1 -- 172.21.15.136:0/202798501 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27200039c0 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272419c800 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f271b7fe700 1 -- 172.21.15.136:0/202798501 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f2720003b20 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f271b7fe700 1 -- 172.21.15.136:0/202798501 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f272001a780 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.816+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272419cca0 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.817+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272410c4c0 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.817+0000 7f271b7fe700 1 -- 172.21.15.136:0/202798501 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f272001a9f0 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.817+0000 7f271b7fe700 1 --2- 172.21.15.136:0/202798501 >> v2:172.21.15.136:6800/1232474076 conn(0x7f2710038e50 0x7f271003b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.818+0000 7f271b7fe700 1 -- 172.21.15.136:0/202798501 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f272004d120 con 0x7f272407bfa0 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.818+0000 7f2729c47700 1 -- 172.21.15.136:0/202798501 >> v2:172.21.15.136:6800/1232474076 conn(0x7f2710038e50 msgr2=0x7f271003b310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/1232474076 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.818+0000 7f2729c47700 1 --2- 172.21.15.136:0/202798501 >> v2:172.21.15.136:6800/1232474076 conn(0x7f2710038e50 0x7f271003b310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T05:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:23.822+0000 7f271b7fe700 1 -- 172.21.15.136:0/202798501 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f272001aca0 con 0x7f272407bfa0 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.019+0000 7f2729c47700 1 -- 172.21.15.136:0/202798501 >> v2:172.21.15.136:6800/1232474076 conn(0x7f2710038e50 msgr2=0x7f271003b310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/1232474076 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.019+0000 7f2729c47700 1 --2- 172.21.15.136:0/202798501 >> v2:172.21.15.136:6800/1232474076 conn(0x7f2710038e50 0x7f271003b310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.077+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f272419f680 con 0x7f272407bfa0 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.078+0000 7f271b7fe700 1 -- 172.21.15.136:0/202798501 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f2720018350 con 0x7f272407bfa0 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 >> v2:172.21.15.136:6800/1232474076 conn(0x7f2710038e50 msgr2=0x7f271003b310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 --2- 172.21.15.136:0/202798501 >> v2:172.21.15.136:6800/1232474076 conn(0x7f2710038e50 0x7f271003b310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 msgr2=0x7f272419c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 --2- 172.21.15.136:0/202798501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272419c0c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f27200046c0 tx=0x7f2720011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 shutdown_connections 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 --2- 172.21.15.136:0/202798501 >> v2:172.21.15.136:6800/1232474076 conn(0x7f2710038e50 0x7f271003b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 --2- 172.21.15.136:0/202798501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f272407bfa0 0x7f272419c0c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 >> 172.21.15.136:0/202798501 conn(0x7f2724101670 msgr2=0x7f2724102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 shutdown_connections 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:24.080+0000 7f272c6ac700 1 -- 172.21.15.136:0/202798501 wait complete. 2024-03-26T05:16:24.588 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for the mgr to restart... 2024-03-26T05:16:24.589 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr epoch 5... 2024-03-26T05:16:24.709 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:24 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:24.387+0000 7f0fc6fc9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T05:16:24.710 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:24 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:24.482+0000 7f0fc6fc9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T05:16:24.710 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:24 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:24.566+0000 7f0fc6fc9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T05:16:24.995 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/202798501' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T05:16:24.995 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:24 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:24.846+0000 7f0fc6fc9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T05:16:24.996 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:24 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:24.957+0000 7f0fc6fc9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T05:16:25.310 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:25 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:25.048+0000 7f0fc6fc9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T05:16:25.310 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:25 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:25.143+0000 7f0fc6fc9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T05:16:25.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:25 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:25.320+0000 7f0fc6fc9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T05:16:26.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:26 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:26.274+0000 7f0fc6fc9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T05:16:26.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:26 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:26.404+0000 7f0fc6fc9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T05:16:27.012 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:26 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:26.738+0000 7f0fc6fc9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T05:16:27.012 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:26 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:26.827+0000 7f0fc6fc9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T05:16:27.012 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:27.021+0000 7f0fc6fc9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:16:27.324 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:27.205+0000 7f0fc6fc9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T05:16:27.740 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:27.489+0000 7f0fc6fc9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:16:28.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:27.750+0000 7f0fc6fc9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T05:16:28.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:27.984+0000 7f0fc6fc9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T05:16:29.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:28 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:28.781+0000 7f0fc6fc9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T05:16:29.649 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:29.349+0000 7f0fc6fc9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T05:16:29.649 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:29.446+0000 7f0fc6fc9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T05:16:29.649 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:29.658+0000 7f0fc6fc9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: Active manager daemon y restarted 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: Activating manager daemon y 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: mgrmap e6: y(active, starting, since 0.00349623s) 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: Manager daemon y is now available 2024-03-26T05:16:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:29 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:30.922 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:30 smithi136 ceph-mon[29560]: Found migration_current of "None". Setting to last migration. 2024-03-26T05:16:30.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:30 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:30.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:30 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:16:30.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:30 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:16:30.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:30 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:30.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:30 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:30.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:30 smithi136 ceph-mon[29560]: mgrmap e7: y(active, since 1.00597s) 2024-03-26T05:16:31.249 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T05:16:31.249 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T05:16:31.249 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.029+0000 7f97a8258700 1 Processor -- start 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.030+0000 7f97a8258700 1 -- start start 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.030+0000 7f97a8258700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0108fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.030+0000 7f97a8258700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a0079f70 con 0x7f97a0106bc0 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.030+0000 7f97a5ff4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0108fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.030+0000 7f97a5ff4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0108fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36910/0 (socket says 172.21.15.136:36910) 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.030+0000 7f97a5ff4700 1 -- 172.21.15.136:0/1998143528 learned_addr learned my addr 172.21.15.136:0/1998143528 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.030+0000 7f97a5ff4700 1 -- 172.21.15.136:0/1998143528 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a01094f0 con 0x7f97a0106bc0 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.030+0000 7f97a5ff4700 1 --2- 172.21.15.136:0/1998143528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0108fb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f979c0098a0 tx=0x7f979c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659dd039212d0bf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a4ff2700 1 -- 172.21.15.136:0/1998143528 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f979c004160 con 0x7f97a0106bc0 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a4ff2700 1 -- 172.21.15.136:0/1998143528 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f979c0042c0 con 0x7f97a0106bc0 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a4ff2700 1 -- 172.21.15.136:0/1998143528 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f979c0119b0 con 0x7f97a0106bc0 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a8258700 1 -- 172.21.15.136:0/1998143528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 msgr2=0x7f97a0108fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a8258700 1 --2- 172.21.15.136:0/1998143528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0108fb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f979c0098a0 tx=0x7f979c009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a8258700 1 -- 172.21.15.136:0/1998143528 shutdown_connections 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a8258700 1 --2- 172.21.15.136:0/1998143528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0108fb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a8258700 1 -- 172.21.15.136:0/1998143528 >> 172.21.15.136:0/1998143528 conn(0x7f97a01005d0 msgr2=0x7f97a01029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a8258700 1 -- 172.21.15.136:0/1998143528 shutdown_connections 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.031+0000 7f97a8258700 1 -- 172.21.15.136:0/1998143528 wait complete. 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.032+0000 7f97a8258700 1 Processor -- start 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.032+0000 7f97a8258700 1 -- start start 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.032+0000 7f97a8258700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0106440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.032+0000 7f97a8258700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a0106980 con 0x7f97a0106bc0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.032+0000 7f97a5ff4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0106440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.032+0000 7f97a5ff4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0106440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36912/0 (socket says 172.21.15.136:36912) 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.032+0000 7f97a5ff4700 1 -- 172.21.15.136:0/1573522400 learned_addr learned my addr 172.21.15.136:0/1573522400 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.033+0000 7f97a5ff4700 1 -- 172.21.15.136:0/1573522400 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c009550 con 0x7f97a0106bc0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.033+0000 7f97a5ff4700 1 --2- 172.21.15.136:0/1573522400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0106440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f979c009130 tx=0x7f979c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.033+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f979c003aa0 con 0x7f97a0106bc0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.033+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f979c01a8c0 con 0x7f97a0106bc0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.033+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a0104b70 con 0x7f97a0106bc0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.033+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f979c019790 con 0x7f97a0106bc0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.033+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a0105010 con 0x7f97a0106bc0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.034+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f979c019a00 con 0x7f97a0106bc0 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.034+0000 7f9796ffd700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 0x7f978c03b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.034+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 --> v2:172.21.15.136:6800/1232474076 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f978c03ba80 con 0x7f978c038ef0 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.034+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f979c04d210 con 0x7f97a0106bc0 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.034+0000 7f97a57f3700 1 -- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 msgr2=0x7f978c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/1232474076 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.034+0000 7f97a57f3700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 0x7f978c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.235+0000 7f97a57f3700 1 -- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 msgr2=0x7f978c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/1232474076 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.235+0000 7f97a57f3700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 0x7f978c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.636+0000 7f97a57f3700 1 -- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 msgr2=0x7f978c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/1232474076 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:25.636+0000 7f97a57f3700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 0x7f978c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:26.437+0000 7f97a57f3700 1 -- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 msgr2=0x7f978c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/1232474076 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:26.437+0000 7f97a57f3700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 0x7f978c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:28.038+0000 7f97a57f3700 1 -- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 msgr2=0x7f978c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/1232474076 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:28.038+0000 7f97a57f3700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 0x7f978c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:29.673+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mgrmap(e 6) v1 ==== 45695+0+0 (secure 0 0 0) 0x7f979c00ed00 con 0x7f97a0106bc0 2024-03-26T05:16:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:29.673+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 msgr2=0x7f978c03b3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:29.673+0000 7f9796ffd700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 0x7f978c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.675+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mgrmap(e 7) v1 ==== 45787+0+0 (secure 0 0 0) 0x7f979c04eef0 con 0x7f97a0106bc0 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.675+0000 7f9796ffd700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/3861450553 conn(0x7f978c03c7c0 0x7f978c03ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.675+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 --> v2:172.21.15.136:6800/3861450553 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f978c03ba80 con 0x7f978c03c7c0 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.675+0000 7f97a57f3700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/3861450553 conn(0x7f978c03c7c0 0x7f978c03ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.676+0000 7f97a57f3700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/3861450553 conn(0x7f978c03c7c0 0x7f978c03ebb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9790003a10 tx=0x7f97900092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.676+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mgr.14118 v2:172.21.15.136:6800/3861450553 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f978c03ba80 con 0x7f978c03c7c0 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 --> v2:172.21.15.136:6800/3861450553 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f97a00636b0 con 0x7f978c03c7c0 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f9796ffd700 1 -- 172.21.15.136:0/1573522400 <== mgr.14118 v2:172.21.15.136:6800/3861450553 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f97a00636b0 con 0x7f978c03c7c0 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/3861450553 conn(0x7f978c03c7c0 msgr2=0x7f978c03ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/3861450553 conn(0x7f978c03c7c0 0x7f978c03ebb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9790003a10 tx=0x7f97900092b0 comp rx=0 tx=0).stop 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 msgr2=0x7f97a0106440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 --2- 172.21.15.136:0/1573522400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0106440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f979c009130 tx=0x7f979c011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 shutdown_connections 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/3861450553 conn(0x7f978c03c7c0 0x7f978c03ebb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 --2- 172.21.15.136:0/1573522400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a0106bc0 0x7f97a0106440 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 --2- 172.21.15.136:0/1573522400 >> v2:172.21.15.136:6800/1232474076 conn(0x7f978c038ef0 0x7f978c03b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 >> 172.21.15.136:0/1573522400 conn(0x7f97a01005d0 msgr2=0x7f97a0101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 shutdown_connections 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:30.683+0000 7f97a8258700 1 -- 172.21.15.136:0/1573522400 wait complete. 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:mgr epoch 5 is available 2024-03-26T05:16:31.255 INFO:teuthology.orchestra.run.smithi136.stdout:Setting orchestrator backend to cephadm... 2024-03-26T05:16:31.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:31 smithi136 ceph-mon[29560]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T05:16:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:31 smithi136 ceph-mon[29560]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T05:16:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:31 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:32.453 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.652+0000 7fd5cae48700 1 Processor -- start 2024-03-26T05:16:32.453 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.652+0000 7fd5cae48700 1 -- start start 2024-03-26T05:16:32.453 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.652+0000 7fd5cae48700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c4106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:32.453 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.652+0000 7fd5cae48700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c4106ae0 con 0x7fd5c41060f0 2024-03-26T05:16:32.453 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.652+0000 7fd5c8be4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c4106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:32.453 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.652+0000 7fd5c8be4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c4106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59984/0 (socket says 172.21.15.136:59984) 2024-03-26T05:16:32.453 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.652+0000 7fd5c8be4700 1 -- 172.21.15.136:0/1838134258 learned_addr learned my addr 172.21.15.136:0/1838134258 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.653+0000 7fd5c8be4700 1 -- 172.21.15.136:0/1838134258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c41072f0 con 0x7fd5c41060f0 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.653+0000 7fd5c8be4700 1 --2- 172.21.15.136:0/1838134258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c4106510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80098a0 tx=0x7fd5b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75af5e8fc216c696 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.653+0000 7fd5c37fe700 1 -- 172.21.15.136:0/1838134258 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5b8004160 con 0x7fd5c41060f0 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.653+0000 7fd5c37fe700 1 -- 172.21.15.136:0/1838134258 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fd5b80042c0 con 0x7fd5c41060f0 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.653+0000 7fd5c37fe700 1 -- 172.21.15.136:0/1838134258 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5b80119b0 con 0x7fd5c41060f0 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 -- 172.21.15.136:0/1838134258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 msgr2=0x7fd5c4106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 --2- 172.21.15.136:0/1838134258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c4106510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80098a0 tx=0x7fd5b8009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 -- 172.21.15.136:0/1838134258 shutdown_connections 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 --2- 172.21.15.136:0/1838134258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c4106510 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 -- 172.21.15.136:0/1838134258 >> 172.21.15.136:0/1838134258 conn(0x7fd5c41016b0 msgr2=0x7fd5c4103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 -- 172.21.15.136:0/1838134258 shutdown_connections 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 -- 172.21.15.136:0/1838134258 wait complete. 2024-03-26T05:16:32.454 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 Processor -- start 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.654+0000 7fd5cae48700 1 -- start start 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5cae48700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5cae48700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c419c600 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5c8be4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5c8be4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c419c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59986/0 (socket says 172.21.15.136:59986) 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5c8be4700 1 -- 172.21.15.136:0/2071422103 learned_addr learned my addr 172.21.15.136:0/2071422103 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5c8be4700 1 -- 172.21.15.136:0/2071422103 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b8009550 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5c8be4700 1 --2- 172.21.15.136:0/2071422103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c419c0c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8006af0 tx=0x7fd5b8011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5c1ffb700 1 -- 172.21.15.136:0/2071422103 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5b80039e0 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5c1ffb700 1 -- 172.21.15.136:0/2071422103 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fd5b8003b40 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c419c800 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.655+0000 7fd5c1ffb700 1 -- 172.21.15.136:0/2071422103 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5b8019720 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.656+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c419cca0 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.656+0000 7fd5c1ffb700 1 -- 172.21.15.136:0/2071422103 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 7) v1 ==== 45787+0+0 (secure 0 0 0) 0x7fd5b8019950 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.656+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c4195df0 con 0x7fd5c41060f0 2024-03-26T05:16:32.455 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.657+0000 7fd5c1ffb700 1 --2- 172.21.15.136:0/2071422103 >> v2:172.21.15.136:6800/3861450553 conn(0x7fd5b4038dd0 0x7fd5b403b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.657+0000 7fd5c1ffb700 1 -- 172.21.15.136:0/2071422103 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fd5b804cca0 con 0x7fd5c41060f0 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.657+0000 7fd5c3fff700 1 --2- 172.21.15.136:0/2071422103 >> v2:172.21.15.136:6800/3861450553 conn(0x7fd5b4038dd0 0x7fd5b403b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.657+0000 7fd5c3fff700 1 --2- 172.21.15.136:0/2071422103 >> v2:172.21.15.136:6800/3861450553 conn(0x7fd5b4038dd0 0x7fd5b403b290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0006fd0 tx=0x7fd5b0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.662+0000 7fd5c1ffb700 1 -- 172.21.15.136:0/2071422103 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd5b8019c00 con 0x7fd5c41060f0 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.725+0000 7fd5c1ffb700 1 -- 172.21.15.136:0/2071422103 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7fd5b802c960 con 0x7fd5c41060f0 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.872+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 --> v2:172.21.15.136:6800/3861450553 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5c40624c0 con 0x7fd5b4038dd0 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.879+0000 7fd5c1ffb700 1 -- 172.21.15.136:0/2071422103 <== mgr.14118 v2:172.21.15.136:6800/3861450553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd5c40624c0 con 0x7fd5b4038dd0 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 >> v2:172.21.15.136:6800/3861450553 conn(0x7fd5b4038dd0 msgr2=0x7fd5b403b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 --2- 172.21.15.136:0/2071422103 >> v2:172.21.15.136:6800/3861450553 conn(0x7fd5b4038dd0 0x7fd5b403b290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0006fd0 tx=0x7fd5b0006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 msgr2=0x7fd5c419c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 --2- 172.21.15.136:0/2071422103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c419c0c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8006af0 tx=0x7fd5b8011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 shutdown_connections 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 --2- 172.21.15.136:0/2071422103 >> v2:172.21.15.136:6800/3861450553 conn(0x7fd5b4038dd0 0x7fd5b403b290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:32.456 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 --2- 172.21.15.136:0/2071422103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c41060f0 0x7fd5c419c0c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:32.457 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 >> 172.21.15.136:0/2071422103 conn(0x7fd5c41016b0 msgr2=0x7fd5c4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:32.457 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 shutdown_connections 2024-03-26T05:16:32.457 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:31.881+0000 7fd5cae48700 1 -- 172.21.15.136:0/2071422103 wait complete. 2024-03-26T05:16:33.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:32 smithi136 ceph-mon[29560]: [26/Mar/2024:05:16:30] ENGINE Bus STARTING 2024-03-26T05:16:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:32 smithi136 ceph-mon[29560]: [26/Mar/2024:05:16:31] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T05:16:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:32 smithi136 ceph-mon[29560]: [26/Mar/2024:05:16:31] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T05:16:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:32 smithi136 ceph-mon[29560]: [26/Mar/2024:05:16:31] ENGINE Bus STARTED 2024-03-26T05:16:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:32 smithi136 ceph-mon[29560]: mgrmap e8: y(active, since 2s) 2024-03-26T05:16:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:32 smithi136 ceph-mon[29560]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:16:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:32 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:32 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.908+0000 7f256672c700 1 Processor -- start 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.908+0000 7f256672c700 1 -- start start 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.908+0000 7f256672c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f2560106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.908+0000 7f256672c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2560106ae0 con 0x7f25601060f0 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.908+0000 7f255ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f2560106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.908+0000 7f255ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f2560106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59988/0 (socket says 172.21.15.136:59988) 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.908+0000 7f255ffff700 1 -- 172.21.15.136:0/1532955394 learned_addr learned my addr 172.21.15.136:0/1532955394 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f255ffff700 1 -- 172.21.15.136:0/1532955394 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25601072f0 con 0x7f25601060f0 2024-03-26T05:16:33.660 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f255ffff700 1 --2- 172.21.15.136:0/1532955394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f2560106510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f25500098a0 tx=0x7f2550009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f1652d2890f49f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f255effd700 1 -- 172.21.15.136:0/1532955394 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2550004160 con 0x7f25601060f0 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f255effd700 1 -- 172.21.15.136:0/1532955394 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f25500042c0 con 0x7f25601060f0 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f255effd700 1 -- 172.21.15.136:0/1532955394 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25500119b0 con 0x7f25601060f0 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f256672c700 1 -- 172.21.15.136:0/1532955394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 msgr2=0x7f2560106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f256672c700 1 --2- 172.21.15.136:0/1532955394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f2560106510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f25500098a0 tx=0x7f2550009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f256672c700 1 -- 172.21.15.136:0/1532955394 shutdown_connections 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f256672c700 1 --2- 172.21.15.136:0/1532955394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f2560106510 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.909+0000 7f256672c700 1 -- 172.21.15.136:0/1532955394 >> 172.21.15.136:0/1532955394 conn(0x7f25601016b0 msgr2=0x7f2560103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.910+0000 7f256672c700 1 -- 172.21.15.136:0/1532955394 shutdown_connections 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.910+0000 7f256672c700 1 -- 172.21.15.136:0/1532955394 wait complete. 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.910+0000 7f256672c700 1 Processor -- start 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.910+0000 7f256672c700 1 -- start start 2024-03-26T05:16:33.661 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.910+0000 7f256672c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f256007bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.910+0000 7f256672c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256007c1b0 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f255ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f256007bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f255ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f256007bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59992/0 (socket says 172.21.15.136:59992) 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f255ffff700 1 -- 172.21.15.136:0/1847592235 learned_addr learned my addr 172.21.15.136:0/1847592235 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f255ffff700 1 -- 172.21.15.136:0/1847592235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2550009550 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f255ffff700 1 --2- 172.21.15.136:0/1847592235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f256007bc70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25500046c0 tx=0x7f2550011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f255d7fa700 1 -- 172.21.15.136:0/1847592235 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25500039c0 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f255d7fa700 1 -- 172.21.15.136:0/1847592235 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f2550003b20 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256007a3a0 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f255d7fa700 1 -- 172.21.15.136:0/1847592235 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f255001a780 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.911+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256007a840 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.912+0000 7f255d7fa700 1 -- 172.21.15.136:0/1847592235 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f255001a8e0 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.912+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25600636b0 con 0x7f25601060f0 2024-03-26T05:16:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.912+0000 7f255d7fa700 1 --2- 172.21.15.136:0/1847592235 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2548038ea0 0x7f254803b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.912+0000 7f255d7fa700 1 -- 172.21.15.136:0/1847592235 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f255004ce70 con 0x7f25601060f0 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.912+0000 7f255f7fe700 1 --2- 172.21.15.136:0/1847592235 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2548038ea0 0x7f254803b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.913+0000 7f255f7fe700 1 --2- 172.21.15.136:0/1847592235 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2548038ea0 0x7f254803b360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2554006fd0 tx=0x7f2554006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:32.917+0000 7f255d7fa700 1 -- 172.21.15.136:0/1847592235 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2550018530 con 0x7f25601060f0 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.111+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 --> v2:172.21.15.136:6800/3861450553 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f2560102930 con 0x7f2548038ea0 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.112+0000 7f255d7fa700 1 -- 172.21.15.136:0/1847592235 <== mgr.14118 v2:172.21.15.136:6800/3861450553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f2560102930 con 0x7f2548038ea0 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2548038ea0 msgr2=0x7f254803b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 --2- 172.21.15.136:0/1847592235 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2548038ea0 0x7f254803b360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2554006fd0 tx=0x7f2554006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 msgr2=0x7f256007bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 --2- 172.21.15.136:0/1847592235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f256007bc70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25500046c0 tx=0x7f2550011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 shutdown_connections 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 --2- 172.21.15.136:0/1847592235 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2548038ea0 0x7f254803b360 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 --2- 172.21.15.136:0/1847592235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25601060f0 0x7f256007bc70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 >> 172.21.15.136:0/1847592235 conn(0x7f25601016b0 msgr2=0x7f2560102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 shutdown_connections 2024-03-26T05:16:33.664 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:33.114+0000 7f256672c700 1 -- 172.21.15.136:0/1847592235 wait complete. 2024-03-26T05:16:33.664 INFO:teuthology.orchestra.run.smithi136.stdout:Generating ssh key... 2024-03-26T05:16:34.290 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:33 smithi136 ceph-mon[29560]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:16:35.048 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: Generating public/private rsa key pair. 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: Your identification has been saved in /tmp/tmphbzauln1/key. 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: Your public key has been saved in /tmp/tmphbzauln1/key.pub. 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: The key fingerprint is: 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: SHA256:sOBPPGsdOnAEnhJ8nggIfBdPxBTGHyS103uhHyBdEy8 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: The key's randomart image is: 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: +---[RSA 3072]----+ 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: |=.. ..*B+o +. | 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: |o..+.++o..+ . o | 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: | .o+=.o..+.+ E . | 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: | .oo+ o .o + o | 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: | o * S o o | 2024-03-26T05:16:35.049 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: | = = . o . | 2024-03-26T05:16:35.050 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: | * . . | 2024-03-26T05:16:35.050 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: | . . | 2024-03-26T05:16:35.050 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: | | 2024-03-26T05:16:35.050 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:34 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: +----[SHA256]-----+ 2024-03-26T05:16:35.331 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.097+0000 7f3169b97700 1 Processor -- start 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.097+0000 7f3169b97700 1 -- start start 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.097+0000 7f3169b97700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316407a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.097+0000 7f3169b97700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316407a9f0 con 0x7f316407bfc0 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.098+0000 7f31637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316407a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.098+0000 7f31637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316407a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60008/0 (socket says 172.21.15.136:60008) 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.098+0000 7f31637fe700 1 -- 172.21.15.136:0/3168681391 learned_addr learned my addr 172.21.15.136:0/3168681391 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.098+0000 7f31637fe700 1 -- 172.21.15.136:0/3168681391 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316407ab30 con 0x7f316407bfc0 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.098+0000 7f31637fe700 1 --2- 172.21.15.136:0/3168681391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316407a420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f31540098a0 tx=0x7f3154009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cbaf53839889405 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.098+0000 7f3162ffd700 1 -- 172.21.15.136:0/3168681391 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3154004160 con 0x7f316407bfc0 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.098+0000 7f3162ffd700 1 -- 172.21.15.136:0/3168681391 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f31540042c0 con 0x7f316407bfc0 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.098+0000 7f3162ffd700 1 -- 172.21.15.136:0/3168681391 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31540119b0 con 0x7f316407bfc0 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.099+0000 7f3169b97700 1 -- 172.21.15.136:0/3168681391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 msgr2=0x7f316407a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.099+0000 7f3169b97700 1 --2- 172.21.15.136:0/3168681391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316407a420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f31540098a0 tx=0x7f3154009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.099+0000 7f3169b97700 1 -- 172.21.15.136:0/3168681391 shutdown_connections 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.099+0000 7f3169b97700 1 --2- 172.21.15.136:0/3168681391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316407a420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.099+0000 7f3169b97700 1 -- 172.21.15.136:0/3168681391 >> 172.21.15.136:0/3168681391 conn(0x7f3164101680 msgr2=0x7f3164103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.099+0000 7f3169b97700 1 -- 172.21.15.136:0/3168681391 shutdown_connections 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.099+0000 7f3169b97700 1 -- 172.21.15.136:0/3168681391 wait complete. 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.099+0000 7f3169b97700 1 Processor -- start 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f3169b97700 1 -- start start 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f3169b97700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316419c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f3169b97700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316419c5d0 con 0x7f316407bfc0 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f31637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316419c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f31637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316419c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60018/0 (socket says 172.21.15.136:60018) 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f31637fe700 1 -- 172.21.15.136:0/3868662908 learned_addr learned my addr 172.21.15.136:0/3868662908 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:35.333 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f31637fe700 1 -- 172.21.15.136:0/3868662908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154009550 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f31637fe700 1 --2- 172.21.15.136:0/3868662908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316419c090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f31540046c0 tx=0x7f3154011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f31617fa700 1 -- 172.21.15.136:0/3868662908 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31540039c0 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f31617fa700 1 -- 172.21.15.136:0/3868662908 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f3154003b20 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316419c7d0 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f31617fa700 1 -- 172.21.15.136:0/3868662908 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f315401a780 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.100+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316419cc70 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.101+0000 7f31617fa700 1 -- 172.21.15.136:0/3868662908 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f315401a9f0 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.101+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316410c490 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.101+0000 7f31617fa700 1 --2- 172.21.15.136:0/3868662908 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3150038e50 0x7f315003b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.101+0000 7f31617fa700 1 -- 172.21.15.136:0/3868662908 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f315404cea0 con 0x7f316407bfc0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.102+0000 7f315bfff700 1 --2- 172.21.15.136:0/3868662908 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3150038e50 0x7f315003b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.102+0000 7f315bfff700 1 --2- 172.21.15.136:0/3868662908 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3150038e50 0x7f315003b310 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f314c006fd0 tx=0x7f314c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:35.334 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.106+0000 7f31617fa700 1 -- 172.21.15.136:0/3868662908 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f315401aca0 con 0x7f316407bfc0 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.299+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 --> v2:172.21.15.136:6800/3861450553 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3164102940 con 0x7f3150038e50 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.752+0000 7f31617fa700 1 -- 172.21.15.136:0/3868662908 <== mgr.14118 v2:172.21.15.136:6800/3861450553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3164102940 con 0x7f3150038e50 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.754+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3150038e50 msgr2=0x7f315003b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.754+0000 7f3169b97700 1 --2- 172.21.15.136:0/3868662908 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3150038e50 0x7f315003b310 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f314c006fd0 tx=0x7f314c006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.754+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 msgr2=0x7f316419c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.754+0000 7f3169b97700 1 --2- 172.21.15.136:0/3868662908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316419c090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f31540046c0 tx=0x7f3154011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.754+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 shutdown_connections 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.754+0000 7f3169b97700 1 --2- 172.21.15.136:0/3868662908 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3150038e50 0x7f315003b310 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.754+0000 7f3169b97700 1 --2- 172.21.15.136:0/3868662908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f316407bfc0 0x7f316419c090 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.754+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 >> 172.21.15.136:0/3868662908 conn(0x7f3164101680 msgr2=0x7f3164102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.755+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 shutdown_connections 2024-03-26T05:16:35.335 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:34.755+0000 7f3169b97700 1 -- 172.21.15.136:0/3868662908 wait complete. 2024-03-26T05:16:35.957 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:35 smithi136 ceph-mon[29560]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:16:35.958 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:35 smithi136 ceph-mon[29560]: Generating ssh key... 2024-03-26T05:16:35.958 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:35 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:35.958 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:35 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:36.630 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNIcTXqe5lI6r5KMXd2SWzQrtf4xWNFn3AqIw6bFsV/h70MymjMjsOBL3TlJXteeCN1DkYmQNK9xluzcbBK073cc0pimIZMAi0wynxUfHiKHAKpFvNT/DT5AnmMq3eN5jwXIvkgH5IiqldelRrLLforTdp4scQYoD5AuS2RD+CS0m+3A9F69pp4JAp8vw9Z2NS1IvkqE3hTdNjQwThL7lqVfuBZ9W7gaaNlgl1WxA+eN0zEKD5+J42SX+W2ZEFQN5WGo32Ks8hifMifncajPJuhYKyEKEAQR+0dmRsOMj7OjvOlj3OC3dLpbG3Sqg6IOWsdgbKN3MpRORNIAdzMQrwPx+0xcZWt41pcrX7EUBIlidwg+JvVpGoi3Op67Vfy/6xM2AuiLLbXPhW9ij25I1VWOhMDzkQJM57Y97+9KUZC/HNOLkz5gqkuYDLEl2901l/u259K77fBTMlAbyDg0cSg07oo+KRQfE5fHnaiQeG4Xx2c1x64ihGmje6rYvQNc= ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:16:36.630 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.765+0000 7f2bcbb75700 1 Processor -- start 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.765+0000 7f2bcbb75700 1 -- start start 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.765+0000 7f2bcbb75700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc4104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.765+0000 7f2bcbb75700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4079f50 con 0x7f2bc41049b0 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.765+0000 7f2bc9911700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc4104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.766+0000 7f2bc9911700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc4104dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60024/0 (socket says 172.21.15.136:60024) 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.766+0000 7f2bc9911700 1 -- 172.21.15.136:0/2289559200 learned_addr learned my addr 172.21.15.136:0/2289559200 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.766+0000 7f2bc9911700 1 -- 172.21.15.136:0/2289559200 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc4105310 con 0x7f2bc41049b0 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.766+0000 7f2bc9911700 1 --2- 172.21.15.136:0/2289559200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc4104dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2bc00098a0 tx=0x7f2bc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945a94fdbc6f6f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.766+0000 7f2bc890f700 1 -- 172.21.15.136:0/2289559200 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bc0004160 con 0x7f2bc41049b0 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.766+0000 7f2bc890f700 1 -- 172.21.15.136:0/2289559200 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f2bc00042c0 con 0x7f2bc41049b0 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.766+0000 7f2bc890f700 1 -- 172.21.15.136:0/2289559200 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bc00119b0 con 0x7f2bc41049b0 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 -- 172.21.15.136:0/2289559200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 msgr2=0x7f2bc4104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 --2- 172.21.15.136:0/2289559200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc4104dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2bc00098a0 tx=0x7f2bc0009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:36.631 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 -- 172.21.15.136:0/2289559200 shutdown_connections 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 --2- 172.21.15.136:0/2289559200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc4104dd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 -- 172.21.15.136:0/2289559200 >> 172.21.15.136:0/2289559200 conn(0x7f2bc41005b0 msgr2=0x7f2bc41029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 -- 172.21.15.136:0/2289559200 shutdown_connections 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 -- 172.21.15.136:0/2289559200 wait complete. 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 Processor -- start 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.767+0000 7f2bcbb75700 1 -- start start 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bcbb75700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc419c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bcbb75700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc419c550 con 0x7f2bc41049b0 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bc9911700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc419c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bc9911700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc419c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60040/0 (socket says 172.21.15.136:60040) 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bc9911700 1 -- 172.21.15.136:0/3578830232 learned_addr learned my addr 172.21.15.136:0/3578830232 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bc9911700 1 -- 172.21.15.136:0/3578830232 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0009550 con 0x7f2bc41049b0 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bc9911700 1 --2- 172.21.15.136:0/3578830232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc419c010 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009130 tx=0x7f2bc0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bbaffd700 1 -- 172.21.15.136:0/3578830232 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bc0003aa0 con 0x7f2bc41049b0 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bbaffd700 1 -- 172.21.15.136:0/3578830232 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f2bc001a8c0 con 0x7f2bc41049b0 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bbaffd700 1 -- 172.21.15.136:0/3578830232 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bc0019790 con 0x7f2bc41049b0 2024-03-26T05:16:36.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc419c750 con 0x7f2bc41049b0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.768+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc419cbf0 con 0x7f2bc41049b0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.769+0000 7f2bbaffd700 1 -- 172.21.15.136:0/3578830232 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f2bc0019a00 con 0x7f2bc41049b0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.769+0000 7f2bbaffd700 1 --2- 172.21.15.136:0/3578830232 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2bb0038ef0 0x7f2bb003b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.769+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc4195ce0 con 0x7f2bc41049b0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.769+0000 7f2bbaffd700 1 -- 172.21.15.136:0/3578830232 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f2bc004cf70 con 0x7f2bc41049b0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.769+0000 7f2bc9110700 1 --2- 172.21.15.136:0/3578830232 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2bb0038ef0 0x7f2bb003b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.770+0000 7f2bc9110700 1 --2- 172.21.15.136:0/3578830232 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2bb0038ef0 0x7f2bb003b3b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4006fd0 tx=0x7f2bb4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.774+0000 7f2bbaffd700 1 -- 172.21.15.136:0/3578830232 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2bc0018c30 con 0x7f2bc41049b0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.967+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 --> v2:172.21.15.136:6800/3861450553 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bc40624c0 con 0x7f2bb0038ef0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.968+0000 7f2bbaffd700 1 -- 172.21.15.136:0/3578830232 <== mgr.14118 v2:172.21.15.136:6800/3861450553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2bc40624c0 con 0x7f2bb0038ef0 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2bb0038ef0 msgr2=0x7f2bb003b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 --2- 172.21.15.136:0/3578830232 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2bb0038ef0 0x7f2bb003b3b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4006fd0 tx=0x7f2bb4006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 msgr2=0x7f2bc419c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 --2- 172.21.15.136:0/3578830232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc419c010 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009130 tx=0x7f2bc0011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 shutdown_connections 2024-03-26T05:16:36.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 --2- 172.21.15.136:0/3578830232 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2bb0038ef0 0x7f2bb003b3b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:36.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 --2- 172.21.15.136:0/3578830232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2bc41049b0 0x7f2bc419c010 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:36.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 >> 172.21.15.136:0/3578830232 conn(0x7f2bc41005b0 msgr2=0x7f2bc4192e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:36.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 shutdown_connections 2024-03-26T05:16:36.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:35.970+0000 7f2bcbb75700 1 -- 172.21.15.136:0/3578830232 wait complete. 2024-03-26T05:16:36.634 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T05:16:36.634 INFO:teuthology.orchestra.run.smithi136.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T05:16:36.634 INFO:teuthology.orchestra.run.smithi136.stdout:Adding host smithi136... 2024-03-26T05:16:37.276 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:36 smithi136 ceph-mon[29560]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:16:38.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:37 smithi136 ceph-mon[29560]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "addr": "172.21.15.136", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:16:39.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:38 smithi136 ceph-mon[29560]: Deploying cephadm binary to smithi136 2024-03-26T05:16:42.128 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Added host 'smithi136' with addr '172.21.15.136' 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.079+0000 7ff903f1a700 1 Processor -- start 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.079+0000 7ff903f1a700 1 -- start start 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.079+0000 7ff903f1a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc07a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.079+0000 7ff903f1a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc07a9b0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.079+0000 7ff901cb6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc07a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.079+0000 7ff901cb6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc07a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60042/0 (socket says 172.21.15.136:60042) 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.079+0000 7ff901cb6700 1 -- 172.21.15.136:0/1883373585 learned_addr learned my addr 172.21.15.136:0/1883373585 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.080+0000 7ff901cb6700 1 -- 172.21.15.136:0/1883373585 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc07aaf0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.080+0000 7ff901cb6700 1 --2- 172.21.15.136:0/1883373585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc07a3e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80098a0 tx=0x7ff8f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1509a340038f8b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.080+0000 7ff900cb4700 1 -- 172.21.15.136:0/1883373585 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8f8004160 con 0x7ff8fc07bf80 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.080+0000 7ff900cb4700 1 -- 172.21.15.136:0/1883373585 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7ff8f80042c0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.129 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.080+0000 7ff900cb4700 1 -- 172.21.15.136:0/1883373585 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8f80119b0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 -- 172.21.15.136:0/1883373585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 msgr2=0x7ff8fc07a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 --2- 172.21.15.136:0/1883373585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc07a3e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80098a0 tx=0x7ff8f8009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 -- 172.21.15.136:0/1883373585 shutdown_connections 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 --2- 172.21.15.136:0/1883373585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc07a3e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 -- 172.21.15.136:0/1883373585 >> 172.21.15.136:0/1883373585 conn(0x7ff8fc1016b0 msgr2=0x7ff8fc103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 -- 172.21.15.136:0/1883373585 shutdown_connections 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 -- 172.21.15.136:0/1883373585 wait complete. 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 Processor -- start 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.081+0000 7ff903f1a700 1 -- start start 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff903f1a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc19c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff903f1a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc19c540 con 0x7ff8fc07bf80 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff901cb6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc19c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:42.130 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff901cb6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc19c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60046/0 (socket says 172.21.15.136:60046) 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff901cb6700 1 -- 172.21.15.136:0/638235489 learned_addr learned my addr 172.21.15.136:0/638235489 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff901cb6700 1 -- 172.21.15.136:0/638235489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f8009550 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff901cb6700 1 --2- 172.21.15.136:0/638235489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc19c000 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009130 tx=0x7ff8f8011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff8f2ffd700 1 -- 172.21.15.136:0/638235489 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8f8003aa0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff8f2ffd700 1 -- 172.21.15.136:0/638235489 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7ff8f801a8c0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc19c740 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff8f2ffd700 1 -- 172.21.15.136:0/638235489 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8f8019790 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.082+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc19cbe0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.083+0000 7ff8f2ffd700 1 -- 172.21.15.136:0/638235489 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7ff8f8019a00 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.083+0000 7ff8f2ffd700 1 --2- 172.21.15.136:0/638235489 >> v2:172.21.15.136:6800/3861450553 conn(0x7ff8e8038ef0 0x7ff8e803b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.083+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc10c4e0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.083+0000 7ff8f2ffd700 1 -- 172.21.15.136:0/638235489 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7ff8f804d0e0 con 0x7ff8fc07bf80 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.083+0000 7ff9014b5700 1 --2- 172.21.15.136:0/638235489 >> v2:172.21.15.136:6800/3861450553 conn(0x7ff8e8038ef0 0x7ff8e803b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.084+0000 7ff9014b5700 1 --2- 172.21.15.136:0/638235489 >> v2:172.21.15.136:6800/3861450553 conn(0x7ff8e8038ef0 0x7ff8e803b3b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec006fd0 tx=0x7ff8ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:42.131 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.088+0000 7ff8f2ffd700 1 -- 172.21.15.136:0/638235489 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff8f8018350 con 0x7ff8fc07bf80 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:37.286+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 --> v2:172.21.15.136:6800/3861450553 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi136", "addr": "172.21.15.136", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8fc102920 con 0x7ff8e8038ef0 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.502+0000 7ff8f2ffd700 1 -- 172.21.15.136:0/638235489 <== mgr.14118 v2:172.21.15.136:6800/3861450553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff8fc102920 con 0x7ff8e8038ef0 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 >> v2:172.21.15.136:6800/3861450553 conn(0x7ff8e8038ef0 msgr2=0x7ff8e803b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 --2- 172.21.15.136:0/638235489 >> v2:172.21.15.136:6800/3861450553 conn(0x7ff8e8038ef0 0x7ff8e803b3b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec006fd0 tx=0x7ff8ec006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 msgr2=0x7ff8fc19c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 --2- 172.21.15.136:0/638235489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc19c000 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009130 tx=0x7ff8f8011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 shutdown_connections 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 --2- 172.21.15.136:0/638235489 >> v2:172.21.15.136:6800/3861450553 conn(0x7ff8e8038ef0 0x7ff8e803b3b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 --2- 172.21.15.136:0/638235489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8fc07bf80 0x7ff8fc19c000 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 >> 172.21.15.136:0/638235489 conn(0x7ff8fc1016b0 msgr2=0x7ff8fc102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 shutdown_connections 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:41.505+0000 7ff903f1a700 1 -- 172.21.15.136:0/638235489 wait complete. 2024-03-26T05:16:42.132 INFO:teuthology.orchestra.run.smithi136.stdout:Deploying unmanaged mon service... 2024-03-26T05:16:42.769 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:42 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:42.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:42 smithi136 ceph-mon[29560]: Added host smithi136 2024-03-26T05:16:42.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:42 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:43.478 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.558+0000 7f2b9b40a700 1 Processor -- start 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.559+0000 7f2b9b40a700 1 -- start start 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.559+0000 7f2b9b40a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b94104e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.559+0000 7f2b9b40a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b94079f30 con 0x7f2b94104a00 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.559+0000 7f2b991a6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b94104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.559+0000 7f2b991a6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b94104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39018/0 (socket says 172.21.15.136:39018) 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.559+0000 7f2b991a6700 1 -- 172.21.15.136:0/4280626049 learned_addr learned my addr 172.21.15.136:0/4280626049 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.559+0000 7f2b991a6700 1 -- 172.21.15.136:0/4280626049 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b94105360 con 0x7f2b94104a00 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.559+0000 7f2b991a6700 1 --2- 172.21.15.136:0/4280626049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b94104e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2b900098a0 tx=0x7f2b90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4bd8185383bd7b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b8bfff700 1 -- 172.21.15.136:0/4280626049 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b90004160 con 0x7f2b94104a00 2024-03-26T05:16:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b8bfff700 1 -- 172.21.15.136:0/4280626049 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f2b900042c0 con 0x7f2b94104a00 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b8bfff700 1 -- 172.21.15.136:0/4280626049 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b900119b0 con 0x7f2b94104a00 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b9b40a700 1 -- 172.21.15.136:0/4280626049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 msgr2=0x7f2b94104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b9b40a700 1 --2- 172.21.15.136:0/4280626049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b94104e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2b900098a0 tx=0x7f2b90009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b9b40a700 1 -- 172.21.15.136:0/4280626049 shutdown_connections 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b9b40a700 1 --2- 172.21.15.136:0/4280626049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b94104e20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b9b40a700 1 -- 172.21.15.136:0/4280626049 >> 172.21.15.136:0/4280626049 conn(0x7f2b941005c0 msgr2=0x7f2b94102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b9b40a700 1 -- 172.21.15.136:0/4280626049 shutdown_connections 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.560+0000 7f2b9b40a700 1 -- 172.21.15.136:0/4280626049 wait complete. 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.561+0000 7f2b9b40a700 1 Processor -- start 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.561+0000 7f2b9b40a700 1 -- start start 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.561+0000 7f2b9b40a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b9419c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.561+0000 7f2b9b40a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9419c560 con 0x7f2b94104a00 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.561+0000 7f2b991a6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b9419c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:43.480 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.561+0000 7f2b991a6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b9419c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39026/0 (socket says 172.21.15.136:39026) 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.561+0000 7f2b991a6700 1 -- 172.21.15.136:0/1849868052 learned_addr learned my addr 172.21.15.136:0/1849868052 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.561+0000 7f2b991a6700 1 -- 172.21.15.136:0/1849868052 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b90009550 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.562+0000 7f2b991a6700 1 --2- 172.21.15.136:0/1849868052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b9419c020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b900046c0 tx=0x7f2b90011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.562+0000 7f2b8a7fc700 1 -- 172.21.15.136:0/1849868052 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b900039c0 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.562+0000 7f2b8a7fc700 1 -- 172.21.15.136:0/1849868052 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f2b90003b20 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.562+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9419c760 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.562+0000 7f2b8a7fc700 1 -- 172.21.15.136:0/1849868052 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b9001a7b0 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.562+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9419cc00 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.563+0000 7f2b8a7fc700 1 -- 172.21.15.136:0/1849868052 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f2b9001aa20 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.563+0000 7f2b8a7fc700 1 --2- 172.21.15.136:0/1849868052 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2b80038e50 0x7f2b8003b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.563+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b94195ce0 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.563+0000 7f2b8a7fc700 1 -- 172.21.15.136:0/1849868052 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f2b9004cea0 con 0x7f2b94104a00 2024-03-26T05:16:43.481 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.563+0000 7f2b989a5700 1 --2- 172.21.15.136:0/1849868052 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2b80038e50 0x7f2b8003b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.563+0000 7f2b989a5700 1 --2- 172.21.15.136:0/1849868052 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2b80038e50 0x7f2b8003b310 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b84006fd0 tx=0x7f2b84006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.568+0000 7f2b8a7fc700 1 -- 172.21.15.136:0/1849868052 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2b900237a0 con 0x7f2b94104a00 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.779+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 --> v2:172.21.15.136:6800/3861450553 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2b940624c0 con 0x7f2b80038e50 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.784+0000 7f2b8a7fc700 1 -- 172.21.15.136:0/1849868052 <== mgr.14118 v2:172.21.15.136:6800/3861450553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2b940624c0 con 0x7f2b80038e50 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2b80038e50 msgr2=0x7f2b8003b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 --2- 172.21.15.136:0/1849868052 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2b80038e50 0x7f2b8003b310 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b84006fd0 tx=0x7f2b84006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 msgr2=0x7f2b9419c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 --2- 172.21.15.136:0/1849868052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b9419c020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b900046c0 tx=0x7f2b90011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 shutdown_connections 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 --2- 172.21.15.136:0/1849868052 >> v2:172.21.15.136:6800/3861450553 conn(0x7f2b80038e50 0x7f2b8003b310 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 --2- 172.21.15.136:0/1849868052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b94104a00 0x7f2b9419c020 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 >> 172.21.15.136:0/1849868052 conn(0x7f2b941005c0 msgr2=0x7f2b94192e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 shutdown_connections 2024-03-26T05:16:43.482 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:42.786+0000 7f2b9b40a700 1 -- 172.21.15.136:0/1849868052 wait complete. 2024-03-26T05:16:43.483 INFO:teuthology.orchestra.run.smithi136.stdout:Deploying unmanaged mgr service... 2024-03-26T05:16:44.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:43 smithi136 ceph-mon[29560]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:16:44.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:43 smithi136 ceph-mon[29560]: Saving service mon spec with placement count:5 2024-03-26T05:16:44.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:43 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.883+0000 7fa80457e700 1 Processor -- start 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.883+0000 7fa80457e700 1 -- start start 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.883+0000 7fa80457e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc105440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.883+0000 7fa80457e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc105a10 con 0x7fa7fc105020 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.884+0000 7fa80231a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc105440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.884+0000 7fa80231a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc105440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39036/0 (socket says 172.21.15.136:39036) 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.884+0000 7fa80231a700 1 -- 172.21.15.136:0/1764079428 learned_addr learned my addr 172.21.15.136:0/1764079428 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:44.711 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.884+0000 7fa80231a700 1 -- 172.21.15.136:0/1764079428 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc106220 con 0x7fa7fc105020 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.884+0000 7fa80231a700 1 --2- 172.21.15.136:0/1764079428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc105440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80098a0 tx=0x7fa7f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5fe14e8205b165 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.884+0000 7fa801318700 1 -- 172.21.15.136:0/1764079428 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7f8004160 con 0x7fa7fc105020 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.884+0000 7fa801318700 1 -- 172.21.15.136:0/1764079428 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fa7f80042c0 con 0x7fa7fc105020 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.884+0000 7fa801318700 1 -- 172.21.15.136:0/1764079428 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7f80119b0 con 0x7fa7fc105020 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 -- 172.21.15.136:0/1764079428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 msgr2=0x7fa7fc105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 --2- 172.21.15.136:0/1764079428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc105440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80098a0 tx=0x7fa7f8009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 -- 172.21.15.136:0/1764079428 shutdown_connections 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 --2- 172.21.15.136:0/1764079428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc105440 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 -- 172.21.15.136:0/1764079428 >> 172.21.15.136:0/1764079428 conn(0x7fa7fc100600 msgr2=0x7fa7fc102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 -- 172.21.15.136:0/1764079428 shutdown_connections 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 -- 172.21.15.136:0/1764079428 wait complete. 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 Processor -- start 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.885+0000 7fa80457e700 1 -- start start 2024-03-26T05:16:44.712 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80457e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80457e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc19a3f0 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80231a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80231a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39046/0 (socket says 172.21.15.136:39046) 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80231a700 1 -- 172.21.15.136:0/3373181561 learned_addr learned my addr 172.21.15.136:0/3373181561 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80231a700 1 -- 172.21.15.136:0/3373181561 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8009550 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80231a700 1 --2- 172.21.15.136:0/3373181561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc199eb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80046c0 tx=0x7fa7f8011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa7f37fe700 1 -- 172.21.15.136:0/3373181561 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7f80039c0 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa7f37fe700 1 -- 172.21.15.136:0/3373181561 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fa7f8003b20 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc19a5f0 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa7f37fe700 1 -- 172.21.15.136:0/3373181561 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7f801a780 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.886+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc19aa90 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.887+0000 7fa7f37fe700 1 -- 172.21.15.136:0/3373181561 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7fa7f8024b50 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.887+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc0636b0 con 0x7fa7fc105020 2024-03-26T05:16:44.713 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.887+0000 7fa7f37fe700 1 --2- 172.21.15.136:0/3373181561 >> v2:172.21.15.136:6800/3861450553 conn(0x7fa7e8038ea0 0x7fa7e803b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.887+0000 7fa7f37fe700 1 -- 172.21.15.136:0/3373181561 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fa7f804cd00 con 0x7fa7fc105020 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.887+0000 7fa801b19700 1 --2- 172.21.15.136:0/3373181561 >> v2:172.21.15.136:6800/3861450553 conn(0x7fa7e8038ea0 0x7fa7e803b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.888+0000 7fa801b19700 1 --2- 172.21.15.136:0/3373181561 >> v2:172.21.15.136:6800/3861450553 conn(0x7fa7e8038ea0 0x7fa7e803b360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec006fd0 tx=0x7fa7ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:43.892+0000 7fa7f37fe700 1 -- 172.21.15.136:0/3373181561 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa7f8024430 con 0x7fa7fc105020 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.108+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 --> v2:172.21.15.136:6800/3861450553 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa7fc101870 con 0x7fa7e8038ea0 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.112+0000 7fa7f37fe700 1 -- 172.21.15.136:0/3373181561 <== mgr.14118 v2:172.21.15.136:6800/3861450553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa7fc101870 con 0x7fa7e8038ea0 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 >> v2:172.21.15.136:6800/3861450553 conn(0x7fa7e8038ea0 msgr2=0x7fa7e803b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 --2- 172.21.15.136:0/3373181561 >> v2:172.21.15.136:6800/3861450553 conn(0x7fa7e8038ea0 0x7fa7e803b360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec006fd0 tx=0x7fa7ec006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 msgr2=0x7fa7fc199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 --2- 172.21.15.136:0/3373181561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc199eb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80046c0 tx=0x7fa7f8011a50 comp rx=0 tx=0).stop 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 shutdown_connections 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 --2- 172.21.15.136:0/3373181561 >> v2:172.21.15.136:6800/3861450553 conn(0x7fa7e8038ea0 0x7fa7e803b360 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 --2- 172.21.15.136:0/3373181561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7fc105020 0x7fa7fc199eb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:44.714 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 >> 172.21.15.136:0/3373181561 conn(0x7fa7fc100600 msgr2=0x7fa7fc101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:44.715 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 shutdown_connections 2024-03-26T05:16:44.715 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:44.114+0000 7fa80457e700 1 -- 172.21.15.136:0/3373181561 wait complete. 2024-03-26T05:16:45.501 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:45 smithi136 ceph-mon[29560]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:16:45.501 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:45 smithi136 ceph-mon[29560]: Saving service mgr spec with placement count:2 2024-03-26T05:16:45.501 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:45 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:46.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:46 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:46 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3219303369' entity='client.admin' 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.068+0000 7f3cae678700 1 Processor -- start 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.068+0000 7f3cae678700 1 -- start start 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.068+0000 7f3cae678700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca8106550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.068+0000 7f3cae678700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8106b20 con 0x7f3ca8106130 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.069+0000 7f3ca7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca8106550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.069+0000 7f3ca7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca8106550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39054/0 (socket says 172.21.15.136:39054) 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.069+0000 7f3ca7fff700 1 -- 172.21.15.136:0/2265430623 learned_addr learned my addr 172.21.15.136:0/2265430623 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.069+0000 7f3ca7fff700 1 -- 172.21.15.136:0/2265430623 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca8107330 con 0x7f3ca8106130 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.069+0000 7f3ca7fff700 1 --2- 172.21.15.136:0/2265430623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca8106550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c980098a0 tx=0x7f3c98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482feb40e33d946f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.069+0000 7f3ca6ffd700 1 -- 172.21.15.136:0/2265430623 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c98004160 con 0x7f3ca8106130 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.069+0000 7f3ca6ffd700 1 -- 172.21.15.136:0/2265430623 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3c980042c0 con 0x7f3ca8106130 2024-03-26T05:16:46.879 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.069+0000 7f3ca6ffd700 1 -- 172.21.15.136:0/2265430623 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c98011960 con 0x7f3ca8106130 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.070+0000 7f3cae678700 1 -- 172.21.15.136:0/2265430623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 msgr2=0x7f3ca8106550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.070+0000 7f3cae678700 1 --2- 172.21.15.136:0/2265430623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca8106550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c980098a0 tx=0x7f3c98009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.070+0000 7f3cae678700 1 -- 172.21.15.136:0/2265430623 shutdown_connections 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.070+0000 7f3cae678700 1 --2- 172.21.15.136:0/2265430623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca8106550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.070+0000 7f3cae678700 1 -- 172.21.15.136:0/2265430623 >> 172.21.15.136:0/2265430623 conn(0x7f3ca81016b0 msgr2=0x7f3ca8103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.070+0000 7f3cae678700 1 -- 172.21.15.136:0/2265430623 shutdown_connections 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.070+0000 7f3cae678700 1 -- 172.21.15.136:0/2265430623 wait complete. 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.071+0000 7f3cae678700 1 Processor -- start 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.071+0000 7f3cae678700 1 -- start start 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.071+0000 7f3cae678700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca807bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.071+0000 7f3cae678700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca807c1b0 con 0x7f3ca8106130 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.071+0000 7f3ca7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca807bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.071+0000 7f3ca7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca807bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39066/0 (socket says 172.21.15.136:39066) 2024-03-26T05:16:46.880 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.071+0000 7f3ca7fff700 1 -- 172.21.15.136:0/3219303369 learned_addr learned my addr 172.21.15.136:0/3219303369 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.071+0000 7f3ca7fff700 1 -- 172.21.15.136:0/3219303369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c98009550 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.072+0000 7f3ca7fff700 1 --2- 172.21.15.136:0/3219303369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca807bc70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3c98006af0 tx=0x7f3c980119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.072+0000 7f3ca57fa700 1 -- 172.21.15.136:0/3219303369 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c980039b0 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.072+0000 7f3ca57fa700 1 -- 172.21.15.136:0/3219303369 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3c98003b10 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.072+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca807a3a0 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.072+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca807a840 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.072+0000 7f3ca57fa700 1 -- 172.21.15.136:0/3219303369 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c98019670 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.073+0000 7f3ca57fa700 1 -- 172.21.15.136:0/3219303369 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f3c98019950 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.073+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca810a560 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.073+0000 7f3ca57fa700 1 --2- 172.21.15.136:0/3219303369 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3c90038e50 0x7f3c9003b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.073+0000 7f3ca57fa700 1 -- 172.21.15.136:0/3219303369 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f3c9804d220 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.073+0000 7f3ca77fe700 1 --2- 172.21.15.136:0/3219303369 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3c90038e50 0x7f3c9003b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.074+0000 7f3ca77fe700 1 --2- 172.21.15.136:0/3219303369 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3c90038e50 0x7f3c9003b310 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c006fd0 tx=0x7f3c9c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.078+0000 7f3ca57fa700 1 -- 172.21.15.136:0/3219303369 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3c98018350 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.279+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f3ca810a770 con 0x7f3ca8106130 2024-03-26T05:16:46.881 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.284+0000 7f3ca57fa700 1 -- 172.21.15.136:0/3219303369 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f3c98023c20 con 0x7f3ca8106130 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3c90038e50 msgr2=0x7f3c9003b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 --2- 172.21.15.136:0/3219303369 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3c90038e50 0x7f3c9003b310 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c006fd0 tx=0x7f3c9c006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 msgr2=0x7f3ca807bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 --2- 172.21.15.136:0/3219303369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca807bc70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3c98006af0 tx=0x7f3c980119e0 comp rx=0 tx=0).stop 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 shutdown_connections 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 --2- 172.21.15.136:0/3219303369 >> v2:172.21.15.136:6800/3861450553 conn(0x7f3c90038e50 0x7f3c9003b310 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 --2- 172.21.15.136:0/3219303369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ca8106130 0x7f3ca807bc70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 >> 172.21.15.136:0/3219303369 conn(0x7f3ca81016b0 msgr2=0x7f3ca8102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:46.882 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 shutdown_connections 2024-03-26T05:16:46.883 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:46.287+0000 7f3cae678700 1 -- 172.21.15.136:0/3219303369 wait complete. 2024-03-26T05:16:48.660 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/296172323' entity='client.admin' 2024-03-26T05:16:48.660 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:48 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:48.667 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.343+0000 7f17aaab5700 1 Processor -- start 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.343+0000 7f17aaab5700 1 -- start start 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.343+0000 7f17aaab5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a4108f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.343+0000 7f17aaab5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a40796c0 con 0x7f17a4106b90 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.344+0000 7f17a8851700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a4108f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.344+0000 7f17a8851700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a4108f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39068/0 (socket says 172.21.15.136:39068) 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.344+0000 7f17a8851700 1 -- 172.21.15.136:0/2025719889 learned_addr learned my addr 172.21.15.136:0/2025719889 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.344+0000 7f17a8851700 1 -- 172.21.15.136:0/2025719889 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a41094c0 con 0x7f17a4106b90 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.344+0000 7f17a8851700 1 --2- 172.21.15.136:0/2025719889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a4108f80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f17980098a0 tx=0x7f1798009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e90c6f8cbad4cc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.344+0000 7f17a37fe700 1 -- 172.21.15.136:0/2025719889 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1798004160 con 0x7f17a4106b90 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.344+0000 7f17a37fe700 1 -- 172.21.15.136:0/2025719889 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f17980042c0 con 0x7f17a4106b90 2024-03-26T05:16:48.668 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.344+0000 7f17a37fe700 1 -- 172.21.15.136:0/2025719889 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1798011960 con 0x7f17a4106b90 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.345+0000 7f17aaab5700 1 -- 172.21.15.136:0/2025719889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 msgr2=0x7f17a4108f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.345+0000 7f17aaab5700 1 --2- 172.21.15.136:0/2025719889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a4108f80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f17980098a0 tx=0x7f1798009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.345+0000 7f17aaab5700 1 -- 172.21.15.136:0/2025719889 shutdown_connections 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.345+0000 7f17aaab5700 1 --2- 172.21.15.136:0/2025719889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a4108f80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.345+0000 7f17aaab5700 1 -- 172.21.15.136:0/2025719889 >> 172.21.15.136:0/2025719889 conn(0x7f17a41005a0 msgr2=0x7f17a41029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.345+0000 7f17aaab5700 1 -- 172.21.15.136:0/2025719889 shutdown_connections 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.345+0000 7f17aaab5700 1 -- 172.21.15.136:0/2025719889 wait complete. 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.346+0000 7f17aaab5700 1 Processor -- start 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.346+0000 7f17aaab5700 1 -- start start 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.346+0000 7f17aaab5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a419c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.346+0000 7f17aaab5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a419c5d0 con 0x7f17a4106b90 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.346+0000 7f17a8851700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a419c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:48.669 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.346+0000 7f17a8851700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a419c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39078/0 (socket says 172.21.15.136:39078) 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.346+0000 7f17a8851700 1 -- 172.21.15.136:0/296172323 learned_addr learned my addr 172.21.15.136:0/296172323 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.346+0000 7f17a8851700 1 -- 172.21.15.136:0/296172323 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1798009550 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.347+0000 7f17a8851700 1 --2- 172.21.15.136:0/296172323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a419c090 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1798004290 tx=0x7f1798011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.347+0000 7f17a1ffb700 1 -- 172.21.15.136:0/296172323 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17980036a0 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.347+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a419c7d0 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.347+0000 7f17a1ffb700 1 -- 172.21.15.136:0/296172323 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f1798024440 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.347+0000 7f17a1ffb700 1 -- 172.21.15.136:0/296172323 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f179801a5c0 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.347+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a419cc70 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.348+0000 7f17a1ffb700 1 -- 172.21.15.136:0/296172323 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f179801a880 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.348+0000 7f17a1ffb700 1 --2- 172.21.15.136:0/296172323 >> v2:172.21.15.136:6800/3861450553 conn(0x7f1794038ea0 0x7f179403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.348+0000 7f17a1ffb700 1 -- 172.21.15.136:0/296172323 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f179804cc00 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.348+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a410a2a0 con 0x7f17a4106b90 2024-03-26T05:16:48.670 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.348+0000 7f17a3fff700 1 --2- 172.21.15.136:0/296172323 >> v2:172.21.15.136:6800/3861450553 conn(0x7f1794038ea0 0x7f179403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.349+0000 7f17a3fff700 1 --2- 172.21.15.136:0/296172323 >> v2:172.21.15.136:6800/3861450553 conn(0x7f1794038ea0 0x7f179403b360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1790006fd0 tx=0x7f1790006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.353+0000 7f17a1ffb700 1 -- 172.21.15.136:0/296172323 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1798018c20 con 0x7f17a4106b90 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.566+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f17a419f5b0 con 0x7f17a4106b90 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.568+0000 7f17a1ffb700 1 -- 172.21.15.136:0/296172323 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f1798018860 con 0x7f17a4106b90 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.570+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 >> v2:172.21.15.136:6800/3861450553 conn(0x7f1794038ea0 msgr2=0x7f179403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.570+0000 7f17aaab5700 1 --2- 172.21.15.136:0/296172323 >> v2:172.21.15.136:6800/3861450553 conn(0x7f1794038ea0 0x7f179403b360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1790006fd0 tx=0x7f1790006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.571+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 msgr2=0x7f17a419c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.571+0000 7f17aaab5700 1 --2- 172.21.15.136:0/296172323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a419c090 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1798004290 tx=0x7f1798011eb0 comp rx=0 tx=0).stop 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.571+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 shutdown_connections 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.571+0000 7f17aaab5700 1 --2- 172.21.15.136:0/296172323 >> v2:172.21.15.136:6800/3861450553 conn(0x7f1794038ea0 0x7f179403b360 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.571+0000 7f17aaab5700 1 --2- 172.21.15.136:0/296172323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17a4106b90 0x7f17a419c090 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:48.671 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.571+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 >> 172.21.15.136:0/296172323 conn(0x7f17a41005a0 msgr2=0x7f17a4101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:48.672 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.571+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 shutdown_connections 2024-03-26T05:16:48.672 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:47.571+0000 7f17aaab5700 1 -- 172.21.15.136:0/296172323 wait complete. 2024-03-26T05:16:48.672 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling the dashboard module... 2024-03-26T05:16:49.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2818051987' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T05:16:49.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:49 smithi136 ceph-mon[29560]: from='mgr.14118 172.21.15.136:0/3379642394' entity='mgr.y' 2024-03-26T05:16:49.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:49 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ignoring --setuser ceph since I am not root 2024-03-26T05:16:49.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:49 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ignoring --setgroup ceph since I am not root 2024-03-26T05:16:50.158 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.129+0000 7fba631a6700 1 Processor -- start 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.129+0000 7fba631a6700 1 -- start start 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.129+0000 7fba631a6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.129+0000 7fba631a6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c106ae0 con 0x7fba5c1060f0 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.129+0000 7fba60f42700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.129+0000 7fba60f42700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39082/0 (socket says 172.21.15.136:39082) 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.129+0000 7fba60f42700 1 -- 172.21.15.136:0/1219114053 learned_addr learned my addr 172.21.15.136:0/1219114053 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.130+0000 7fba60f42700 1 -- 172.21.15.136:0/1219114053 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c1072f0 con 0x7fba5c1060f0 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.130+0000 7fba60f42700 1 --2- 172.21.15.136:0/1219114053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c106510 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba500098a0 tx=0x7fba50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5fa1cf21757a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.130+0000 7fba5b7fe700 1 -- 172.21.15.136:0/1219114053 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba50004160 con 0x7fba5c1060f0 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.130+0000 7fba5b7fe700 1 -- 172.21.15.136:0/1219114053 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fba500042c0 con 0x7fba5c1060f0 2024-03-26T05:16:50.159 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.130+0000 7fba5b7fe700 1 -- 172.21.15.136:0/1219114053 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba50011960 con 0x7fba5c1060f0 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 -- 172.21.15.136:0/1219114053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 msgr2=0x7fba5c106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 --2- 172.21.15.136:0/1219114053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c106510 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fba500098a0 tx=0x7fba50009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 -- 172.21.15.136:0/1219114053 shutdown_connections 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 --2- 172.21.15.136:0/1219114053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c106510 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 -- 172.21.15.136:0/1219114053 >> 172.21.15.136:0/1219114053 conn(0x7fba5c101690 msgr2=0x7fba5c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 -- 172.21.15.136:0/1219114053 shutdown_connections 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 -- 172.21.15.136:0/1219114053 wait complete. 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 Processor -- start 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.131+0000 7fba631a6700 1 -- start start 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba631a6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c07b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba631a6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c07b920 con 0x7fba5c1060f0 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba60f42700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c07b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba60f42700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c07b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39084/0 (socket says 172.21.15.136:39084) 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba60f42700 1 -- 172.21.15.136:0/2818051987 learned_addr learned my addr 172.21.15.136:0/2818051987 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:50.160 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba60f42700 1 -- 172.21.15.136:0/2818051987 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50009550 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba60f42700 1 --2- 172.21.15.136:0/2818051987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c07b3e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fba50006af0 tx=0x7fba500119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba59ffb700 1 -- 172.21.15.136:0/2818051987 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba500039b0 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba59ffb700 1 -- 172.21.15.136:0/2818051987 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fba50003b10 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c079b10 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.132+0000 7fba59ffb700 1 -- 172.21.15.136:0/2818051987 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba50019670 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.133+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c079fb0 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.133+0000 7fba59ffb700 1 -- 172.21.15.136:0/2818051987 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7fba500198e0 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.134+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c10a520 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.134+0000 7fba59ffb700 1 --2- 172.21.15.136:0/2818051987 >> v2:172.21.15.136:6800/3861450553 conn(0x7fba4c038ea0 0x7fba4c03b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.134+0000 7fba59ffb700 1 -- 172.21.15.136:0/2818051987 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fba5004dff0 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.134+0000 7fba5bfff700 1 --2- 172.21.15.136:0/2818051987 >> v2:172.21.15.136:6800/3861450553 conn(0x7fba4c038ea0 0x7fba4c03b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.134+0000 7fba5bfff700 1 --2- 172.21.15.136:0/2818051987 >> v2:172.21.15.136:6800/3861450553 conn(0x7fba4c038ea0 0x7fba4c03b360 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fba48006fd0 tx=0x7fba48006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.139+0000 7fba59ffb700 1 -- 172.21.15.136:0/2818051987 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fba50023e00 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.403+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fba5c10a730 con 0x7fba5c1060f0 2024-03-26T05:16:50.161 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.577+0000 7fba59ffb700 1 -- 172.21.15.136:0/2818051987 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fba50024e40 con 0x7fba5c1060f0 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.578+0000 7fba59ffb700 1 -- 172.21.15.136:0/2818051987 <== mon.0 v2:172.21.15.136:3300/0 8 ==== mgrmap(e 9) v1 ==== 45906+0+0 (secure 0 0 0) 0x7fba5004ce10 con 0x7fba5c1060f0 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 >> v2:172.21.15.136:6800/3861450553 conn(0x7fba4c038ea0 msgr2=0x7fba4c03b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 --2- 172.21.15.136:0/2818051987 >> v2:172.21.15.136:6800/3861450553 conn(0x7fba4c038ea0 0x7fba4c03b360 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fba48006fd0 tx=0x7fba48006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 msgr2=0x7fba5c07b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 --2- 172.21.15.136:0/2818051987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c07b3e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fba50006af0 tx=0x7fba500119e0 comp rx=0 tx=0).stop 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 shutdown_connections 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 --2- 172.21.15.136:0/2818051987 >> v2:172.21.15.136:6800/3861450553 conn(0x7fba4c038ea0 0x7fba4c03b360 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fba48006fd0 tx=0x7fba48006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 --2- 172.21.15.136:0/2818051987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba5c1060f0 0x7fba5c07b3e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 >> 172.21.15.136:0/2818051987 conn(0x7fba5c101690 msgr2=0x7fba5c102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 shutdown_connections 2024-03-26T05:16:50.162 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:49.580+0000 7fba631a6700 1 -- 172.21.15.136:0/2818051987 wait complete. 2024-03-26T05:16:50.242 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:50 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:50.147+0000 7f39eac28200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T05:16:50.242 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:50 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:50.251+0000 7f39eac28200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T05:16:50.809 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2818051987' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T05:16:50.809 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:50 smithi136 ceph-mon[29560]: mgrmap e9: y(active, since 19s) 2024-03-26T05:16:51.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:50 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:50.918+0000 7f39eac28200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T05:16:51.074 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:51.011+0000 7f39eac28200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T05:16:51.548 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:51.154+0000 7f39eac28200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T05:16:51.549 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:51.252+0000 7f39eac28200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T05:16:51.549 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:51.336+0000 7f39eac28200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.613+0000 7fac23217700 1 Processor -- start 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.613+0000 7fac23217700 1 -- start start 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.613+0000 7fac23217700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:51.559 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.613+0000 7fac23217700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c079f70 con 0x7fac1c106b80 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.613+0000 7fac20fb3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.613+0000 7fac20fb3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39118/0 (socket says 172.21.15.136:39118) 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.613+0000 7fac20fb3700 1 -- 172.21.15.136:0/924102462 learned_addr learned my addr 172.21.15.136:0/924102462 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.613+0000 7fac20fb3700 1 -- 172.21.15.136:0/924102462 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c1094b0 con 0x7fac1c106b80 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.614+0000 7fac20fb3700 1 --2- 172.21.15.136:0/924102462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c108f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fac100098a0 tx=0x7fac10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9308bf56fa733683 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.614+0000 7fac1b7fe700 1 -- 172.21.15.136:0/924102462 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac10004160 con 0x7fac1c106b80 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.614+0000 7fac1b7fe700 1 -- 172.21.15.136:0/924102462 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fac100042c0 con 0x7fac1c106b80 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.614+0000 7fac1b7fe700 1 -- 172.21.15.136:0/924102462 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac10011960 con 0x7fac1c106b80 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.615+0000 7fac23217700 1 -- 172.21.15.136:0/924102462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 msgr2=0x7fac1c108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.615+0000 7fac23217700 1 --2- 172.21.15.136:0/924102462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c108f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fac100098a0 tx=0x7fac10009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.615+0000 7fac23217700 1 -- 172.21.15.136:0/924102462 shutdown_connections 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.615+0000 7fac23217700 1 --2- 172.21.15.136:0/924102462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c108f70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.615+0000 7fac23217700 1 -- 172.21.15.136:0/924102462 >> 172.21.15.136:0/924102462 conn(0x7fac1c100550 msgr2=0x7fac1c1029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.615+0000 7fac23217700 1 -- 172.21.15.136:0/924102462 shutdown_connections 2024-03-26T05:16:51.560 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.615+0000 7fac23217700 1 -- 172.21.15.136:0/924102462 wait complete. 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.615+0000 7fac23217700 1 Processor -- start 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac23217700 1 -- start start 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac23217700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac23217700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c19a360 con 0x7fac1c106b80 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac20fb3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac20fb3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39122/0 (socket says 172.21.15.136:39122) 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac20fb3700 1 -- 172.21.15.136:0/1541601775 learned_addr learned my addr 172.21.15.136:0/1541601775 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac20fb3700 1 -- 172.21.15.136:0/1541601775 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10009550 con 0x7fac1c106b80 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac20fb3700 1 --2- 172.21.15.136:0/1541601775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c199e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac10006af0 tx=0x7fac100119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac19ffb700 1 -- 172.21.15.136:0/1541601775 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac100039b0 con 0x7fac1c106b80 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac19ffb700 1 -- 172.21.15.136:0/1541601775 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fac10003b10 con 0x7fac1c106b80 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c19a560 con 0x7fac1c106b80 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.616+0000 7fac19ffb700 1 -- 172.21.15.136:0/1541601775 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac10019680 con 0x7fac1c106b80 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.617+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c19aa00 con 0x7fac1c106b80 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.618+0000 7fac19ffb700 1 -- 172.21.15.136:0/1541601775 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 9) v1 ==== 45906+0+0 (secure 0 0 0) 0x7fac100198f0 con 0x7fac1c106b80 2024-03-26T05:16:51.561 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.618+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c10a290 con 0x7fac1c106b80 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.618+0000 7fac19ffb700 1 --2- 172.21.15.136:0/1541601775 >> v2:172.21.15.136:6800/3861450553 conn(0x7fac0c038ef0 0x7fac0c03b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.618+0000 7fac19ffb700 1 -- 172.21.15.136:0/1541601775 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fac1004d130 con 0x7fac1c106b80 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.618+0000 7fac1bfff700 1 -- 172.21.15.136:0/1541601775 >> v2:172.21.15.136:6800/3861450553 conn(0x7fac0c038ef0 msgr2=0x7fac0c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/3861450553 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.618+0000 7fac1bfff700 1 --2- 172.21.15.136:0/1541601775 >> v2:172.21.15.136:6800/3861450553 conn(0x7fac0c038ef0 0x7fac0c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.627+0000 7fac19ffb700 1 -- 172.21.15.136:0/1541601775 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fac10018d70 con 0x7fac1c106b80 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.819+0000 7fac1bfff700 1 -- 172.21.15.136:0/1541601775 >> v2:172.21.15.136:6800/3861450553 conn(0x7fac0c038ef0 msgr2=0x7fac0c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/3861450553 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.819+0000 7fac1bfff700 1 --2- 172.21.15.136:0/1541601775 >> v2:172.21.15.136:6800/3861450553 conn(0x7fac0c038ef0 0x7fac0c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.884+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fac1c19d2e0 con 0x7fac1c106b80 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.885+0000 7fac19ffb700 1 -- 172.21.15.136:0/1541601775 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fac10018b90 con 0x7fac1c106b80 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 >> v2:172.21.15.136:6800/3861450553 conn(0x7fac0c038ef0 msgr2=0x7fac0c03b3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 --2- 172.21.15.136:0/1541601775 >> v2:172.21.15.136:6800/3861450553 conn(0x7fac0c038ef0 0x7fac0c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 msgr2=0x7fac1c199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:51.562 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 --2- 172.21.15.136:0/1541601775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c199e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac10006af0 tx=0x7fac100119e0 comp rx=0 tx=0).stop 2024-03-26T05:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 shutdown_connections 2024-03-26T05:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 --2- 172.21.15.136:0/1541601775 >> v2:172.21.15.136:6800/3861450553 conn(0x7fac0c038ef0 0x7fac0c03b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 --2- 172.21.15.136:0/1541601775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac1c106b80 0x7fac1c199e20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 >> 172.21.15.136:0/1541601775 conn(0x7fac1c100550 msgr2=0x7fac1c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 shutdown_connections 2024-03-26T05:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:50.887+0000 7fac23217700 1 -- 172.21.15.136:0/1541601775 wait complete. 2024-03-26T05:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for the mgr to restart... 2024-03-26T05:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr epoch 9... 2024-03-26T05:16:51.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1541601775' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T05:16:51.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:51.607+0000 7f39eac28200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T05:16:51.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:51.719+0000 7f39eac28200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T05:16:51.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:51.807+0000 7f39eac28200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T05:16:52.083 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:51 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:51.909+0000 7f39eac28200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T05:16:52.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:52 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:52.092+0000 7f39eac28200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T05:16:53.323 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:53 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:53.042+0000 7f39eac28200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T05:16:53.324 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:53 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:53.172+0000 7f39eac28200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T05:16:53.789 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:53 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:53.514+0000 7f39eac28200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T05:16:53.789 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:53 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:53.603+0000 7f39eac28200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T05:16:53.790 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:53 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:53.799+0000 7f39eac28200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:16:54.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:53 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:53.983+0000 7f39eac28200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T05:16:54.517 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:54 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:54.267+0000 7f39eac28200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:16:54.517 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:54 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:54.527+0000 7f39eac28200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T05:16:54.823 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:54 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:54.764+0000 7f39eac28200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T05:16:55.823 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:55 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:55.569+0000 7f39eac28200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T05:16:56.475 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:56.166+0000 7f39eac28200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T05:16:56.476 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:56.267+0000 7f39eac28200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T05:16:56.476 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:16:56.485+0000 7f39eac28200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T05:16:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: Active manager daemon y restarted 2024-03-26T05:16:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: Activating manager daemon y 2024-03-26T05:16:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T05:16:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: mgrmap e10: y(active, starting, since 0.00353879s) 2024-03-26T05:16:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:16:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:16:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:16:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:16:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:16:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: Manager daemon y is now available 2024-03-26T05:16:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:16:57.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:16:57.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:16:57.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:57 smithi136 ceph-mon[29560]: mgrmap e11: y(active, since 1.00621s) 2024-03-26T05:16:58.074 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-03-26T05:16:58.074 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T05:16:58.074 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T05:16:58.074 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-03-26T05:16:58.074 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.048+0000 7f8a51a4d700 1 Processor -- start 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.048+0000 7f8a51a4d700 1 -- start start 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.048+0000 7f8a51a4d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c07a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.048+0000 7f8a51a4d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c07a990 con 0x7f8a4c07bf60 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.048+0000 7f8a4affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c07a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.048+0000 7f8a4affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c07a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39638/0 (socket says 172.21.15.136:39638) 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.048+0000 7f8a4affd700 1 -- 172.21.15.136:0/3558463666 learned_addr learned my addr 172.21.15.136:0/3558463666 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.049+0000 7f8a4affd700 1 -- 172.21.15.136:0/3558463666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c07aad0 con 0x7f8a4c07bf60 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.049+0000 7f8a4affd700 1 --2- 172.21.15.136:0/3558463666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c07a3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8a400098a0 tx=0x7f8a40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ceb142907805a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.049+0000 7f8a49ffb700 1 -- 172.21.15.136:0/3558463666 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a40004160 con 0x7f8a4c07bf60 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.049+0000 7f8a49ffb700 1 -- 172.21.15.136:0/3558463666 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f8a400042c0 con 0x7f8a4c07bf60 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.049+0000 7f8a49ffb700 1 -- 172.21.15.136:0/3558463666 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a40011960 con 0x7f8a4c07bf60 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 -- 172.21.15.136:0/3558463666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 msgr2=0x7f8a4c07a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 --2- 172.21.15.136:0/3558463666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c07a3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8a400098a0 tx=0x7f8a40009bb0 comp rx=0 tx=0).stop 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 -- 172.21.15.136:0/3558463666 shutdown_connections 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 --2- 172.21.15.136:0/3558463666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c07a3c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 -- 172.21.15.136:0/3558463666 >> 172.21.15.136:0/3558463666 conn(0x7f8a4c101670 msgr2=0x7f8a4c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 -- 172.21.15.136:0/3558463666 shutdown_connections 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 -- 172.21.15.136:0/3558463666 wait complete. 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 Processor -- start 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.050+0000 7f8a51a4d700 1 -- start start 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a51a4d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c19c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a51a4d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c19c5d0 con 0x7f8a4c07bf60 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a4affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c19c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a4affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c19c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39644/0 (socket says 172.21.15.136:39644) 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a4affd700 1 -- 172.21.15.136:0/1669805869 learned_addr learned my addr 172.21.15.136:0/1669805869 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a4affd700 1 -- 172.21.15.136:0/1669805869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a40009550 con 0x7f8a4c07bf60 2024-03-26T05:16:58.076 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a4affd700 1 --2- 172.21.15.136:0/1669805869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c19c090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8a40004290 tx=0x7f8a40011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a400036a0 con 0x7f8a4c07bf60 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f8a40024440 con 0x7f8a4c07bf60 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.051+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c19c7d0 con 0x7f8a4c07bf60 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.052+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4001a5c0 con 0x7f8a4c07bf60 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.052+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c19cc70 con 0x7f8a4c07bf60 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.052+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 9) v1 ==== 45906+0+0 (secure 0 0 0) 0x7f8a4001a830 con 0x7f8a4c07bf60 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.052+0000 7f8a50a4b700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 0x7f8a3403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.053+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f8a400248b0 con 0x7f8a4c07bf60 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.053+0000 7f8a4a7fc700 1 -- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 msgr2=0x7f8a3403b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/3861450553 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.053+0000 7f8a4a7fc700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 0x7f8a3403b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.053+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 --> v2:172.21.15.136:6800/3861450553 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a4c0636b0 con 0x7f8a34038ba0 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.254+0000 7f8a4a7fc700 1 -- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 msgr2=0x7f8a3403b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/3861450553 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.254+0000 7f8a4a7fc700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 0x7f8a3403b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T05:16:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.655+0000 7f8a4a7fc700 1 -- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 msgr2=0x7f8a3403b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/3861450553 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:52.655+0000 7f8a4a7fc700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 0x7f8a3403b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:53.456+0000 7f8a4a7fc700 1 -- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 msgr2=0x7f8a3403b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/3861450553 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:53.456+0000 7f8a4a7fc700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 0x7f8a3403b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:55.057+0000 7f8a4a7fc700 1 -- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 msgr2=0x7f8a3403b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/3861450553 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:55.057+0000 7f8a4a7fc700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 0x7f8a3403b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:56.500+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mgrmap(e 10) v1 ==== 45708+0+0 (secure 0 0 0) 0x7f8a400245b0 con 0x7f8a4c07bf60 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:56.500+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 msgr2=0x7f8a3403b060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:56.500+0000 7f8a50a4b700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 0x7f8a3403b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.501+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mgrmap(e 11) v1 ==== 45800+0+0 (secure 0 0 0) 0x7f8a40016d20 con 0x7f8a4c07bf60 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.502+0000 7f8a50a4b700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8a3403c4c0 0x7f8a3403e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.502+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 --> v2:172.21.15.136:6800/2365933975 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a4c0636b0 con 0x7f8a3403c4c0 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.503+0000 7f8a4a7fc700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8a3403c4c0 0x7f8a3403e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.503+0000 7f8a4a7fc700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8a3403c4c0 0x7f8a3403e8b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c003a10 tx=0x7f8a3c0092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:58.078 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.504+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f8a4c0636b0 con 0x7f8a3403c4c0 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.511+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 --> v2:172.21.15.136:6800/2365933975 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8a4c10c4f0 con 0x7f8a3403c4c0 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.511+0000 7f8a50a4b700 1 -- 172.21.15.136:0/1669805869 <== mgr.14150 v2:172.21.15.136:6800/2365933975 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8a4c10c4f0 con 0x7f8a3403c4c0 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.511+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8a3403c4c0 msgr2=0x7f8a3403e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.511+0000 7f8a51a4d700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8a3403c4c0 0x7f8a3403e8b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c003a10 tx=0x7f8a3c0092b0 comp rx=0 tx=0).stop 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.511+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 msgr2=0x7f8a4c19c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.511+0000 7f8a51a4d700 1 --2- 172.21.15.136:0/1669805869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c19c090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8a40004290 tx=0x7f8a40011eb0 comp rx=0 tx=0).stop 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.512+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 shutdown_connections 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.512+0000 7f8a51a4d700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8a3403c4c0 0x7f8a3403e8b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.512+0000 7f8a51a4d700 1 --2- 172.21.15.136:0/1669805869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a4c07bf60 0x7f8a4c19c090 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.512+0000 7f8a51a4d700 1 --2- 172.21.15.136:0/1669805869 >> v2:172.21.15.136:6800/3861450553 conn(0x7f8a34038ba0 0x7f8a3403b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.512+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 >> 172.21.15.136:0/1669805869 conn(0x7f8a4c101670 msgr2=0x7f8a4c1022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.512+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 shutdown_connections 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:57.512+0000 7f8a51a4d700 1 -- 172.21.15.136:0/1669805869 wait complete. 2024-03-26T05:16:58.079 INFO:teuthology.orchestra.run.smithi136.stdout:mgr epoch 9 is available 2024-03-26T05:16:58.080 INFO:teuthology.orchestra.run.smithi136.stdout:Generating a dashboard self-signed certificate... 2024-03-26T05:16:58.703 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:58 smithi136 ceph-mon[29560]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T05:16:58.703 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:58 smithi136 ceph-mon[29560]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T05:16:59.372 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.500+0000 7ff806bc8700 1 Processor -- start 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.500+0000 7ff806bc8700 1 -- start start 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.500+0000 7ff806bc8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800106500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.500+0000 7ff806bc8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff800106ad0 con 0x7ff8001060e0 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.501+0000 7ff804964700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.501+0000 7ff804964700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39716/0 (socket says 172.21.15.136:39716) 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.501+0000 7ff804964700 1 -- 172.21.15.136:0/78958553 learned_addr learned my addr 172.21.15.136:0/78958553 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.501+0000 7ff804964700 1 -- 172.21.15.136:0/78958553 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8001072e0 con 0x7ff8001060e0 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.501+0000 7ff804964700 1 --2- 172.21.15.136:0/78958553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800106500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009e80 tx=0x7ff7f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25dc52f6b07928 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.501+0000 7ff7ff7fe700 1 -- 172.21.15.136:0/78958553 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7f800e070 con 0x7ff8001060e0 2024-03-26T05:16:59.373 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.501+0000 7ff7ff7fe700 1 -- 172.21.15.136:0/78958553 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ff7f800b8c0 con 0x7ff8001060e0 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.501+0000 7ff7ff7fe700 1 -- 172.21.15.136:0/78958553 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7f80043d0 con 0x7ff8001060e0 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.502+0000 7ff806bc8700 1 -- 172.21.15.136:0/78958553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 msgr2=0x7ff800106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.502+0000 7ff806bc8700 1 --2- 172.21.15.136:0/78958553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800106500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8009e80 tx=0x7ff7f800b330 comp rx=0 tx=0).stop 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.502+0000 7ff806bc8700 1 -- 172.21.15.136:0/78958553 shutdown_connections 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.502+0000 7ff806bc8700 1 --2- 172.21.15.136:0/78958553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800106500 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.502+0000 7ff806bc8700 1 -- 172.21.15.136:0/78958553 >> 172.21.15.136:0/78958553 conn(0x7ff800101680 msgr2=0x7ff800103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.502+0000 7ff806bc8700 1 -- 172.21.15.136:0/78958553 shutdown_connections 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.502+0000 7ff806bc8700 1 -- 172.21.15.136:0/78958553 wait complete. 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff806bc8700 1 Processor -- start 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff806bc8700 1 -- start start 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff806bc8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff806bc8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80019a3a0 con 0x7ff8001060e0 2024-03-26T05:16:59.374 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff804964700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff804964700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39720/0 (socket says 172.21.15.136:39720) 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff804964700 1 -- 172.21.15.136:0/2938633725 learned_addr learned my addr 172.21.15.136:0/2938633725 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff804964700 1 -- 172.21.15.136:0/2938633725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8009b30 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff804964700 1 --2- 172.21.15.136:0/2938633725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800199e60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8000f80 tx=0x7ff7f8003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff7fdffb700 1 -- 172.21.15.136:0/2938633725 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7f800e040 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff7fdffb700 1 -- 172.21.15.136:0/2938633725 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ff7f800bd30 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80019a5a0 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.503+0000 7ff7fdffb700 1 -- 172.21.15.136:0/2938633725 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7f80126d0 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.504+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80019aaa0 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.504+0000 7ff7fdffb700 1 -- 172.21.15.136:0/2938633725 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 11) v1 ==== 45800+0+0 (secure 0 0 0) 0x7ff7f801c650 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.504+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8000636b0 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.505+0000 7ff7fdffb700 1 --2- 172.21.15.136:0/2938633725 >> v2:172.21.15.136:6800/2365933975 conn(0x7ff7ec038dd0 0x7ff7ec03b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.505+0000 7ff7fdffb700 1 -- 172.21.15.136:0/2938633725 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7ff7f804cee0 con 0x7ff8001060e0 2024-03-26T05:16:59.375 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.505+0000 7ff7fffff700 1 --2- 172.21.15.136:0/2938633725 >> v2:172.21.15.136:6800/2365933975 conn(0x7ff7ec038dd0 0x7ff7ec03b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.505+0000 7ff7fffff700 1 --2- 172.21.15.136:0/2938633725 >> v2:172.21.15.136:6800/2365933975 conn(0x7ff7ec038dd0 0x7ff7ec03b290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8006fd0 tx=0x7ff7e8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.509+0000 7ff7fdffb700 1 -- 172.21.15.136:0/2938633725 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff7f8011390 con 0x7ff8001060e0 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.554+0000 7ff7fdffb700 1 -- 172.21.15.136:0/2938633725 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mgrmap(e 12) v1 ==== 45906+0+0 (secure 0 0 0) 0x7ff7f8011dc0 con 0x7ff8001060e0 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.712+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8001028d0 con 0x7ff7ec038dd0 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.816+0000 7ff7fdffb700 1 -- 172.21.15.136:0/2938633725 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7ff8001028d0 con 0x7ff7ec038dd0 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 >> v2:172.21.15.136:6800/2365933975 conn(0x7ff7ec038dd0 msgr2=0x7ff7ec03b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 --2- 172.21.15.136:0/2938633725 >> v2:172.21.15.136:6800/2365933975 conn(0x7ff7ec038dd0 0x7ff7ec03b290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8006fd0 tx=0x7ff7e8006e40 comp rx=0 tx=0).stop 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 msgr2=0x7ff800199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 --2- 172.21.15.136:0/2938633725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800199e60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8000f80 tx=0x7ff7f8003d00 comp rx=0 tx=0).stop 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 shutdown_connections 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 --2- 172.21.15.136:0/2938633725 >> v2:172.21.15.136:6800/2365933975 conn(0x7ff7ec038dd0 0x7ff7ec03b290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 --2- 172.21.15.136:0/2938633725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8001060e0 0x7ff800199e60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 >> 172.21.15.136:0/2938633725 conn(0x7ff800101680 msgr2=0x7ff8001021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 shutdown_connections 2024-03-26T05:16:59.376 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:58.818+0000 7ff806bc8700 1 -- 172.21.15.136:0/2938633725 wait complete. 2024-03-26T05:16:59.377 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial admin user... 2024-03-26T05:16:59.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:59 smithi136 ceph-mon[29560]: [26/Mar/2024:05:16:57] ENGINE Bus STARTING 2024-03-26T05:16:59.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:59 smithi136 ceph-mon[29560]: [26/Mar/2024:05:16:57] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T05:16:59.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:59 smithi136 ceph-mon[29560]: [26/Mar/2024:05:16:57] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T05:16:59.627 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:59 smithi136 ceph-mon[29560]: [26/Mar/2024:05:16:57] ENGINE Bus STARTED 2024-03-26T05:16:59.627 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:59 smithi136 ceph-mon[29560]: mgrmap e12: y(active, since 2s) 2024-03-26T05:16:59.627 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:16:59.627 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:16:59.627 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:16:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:00.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:00 smithi136 ceph-mon[29560]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:00.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:00.971 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$FXvUYaXTjdJvYZs3f0VfqeMrQuRpAEO66zB4jpI0RBxBaYGLvewzS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711430220, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T05:17:00.971 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.848+0000 7fd69f287700 1 Processor -- start 2024-03-26T05:17:00.971 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.848+0000 7fd69f287700 1 -- start start 2024-03-26T05:17:00.971 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd69f287700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:00.971 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd69f287700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6981059f0 con 0x7fd698105000 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd69d023700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd69d023700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39734/0 (socket says 172.21.15.136:39734) 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd69d023700 1 -- 172.21.15.136:0/2403915697 learned_addr learned my addr 172.21.15.136:0/2403915697 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd69d023700 1 -- 172.21.15.136:0/2403915697 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698106200 con 0x7fd698105000 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd69d023700 1 --2- 172.21.15.136:0/2403915697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698105420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd6880098a0 tx=0x7fd688009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fd0bef584159f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd68ffff700 1 -- 172.21.15.136:0/2403915697 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd688004160 con 0x7fd698105000 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd68ffff700 1 -- 172.21.15.136:0/2403915697 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fd6880042c0 con 0x7fd698105000 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.849+0000 7fd68ffff700 1 -- 172.21.15.136:0/2403915697 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd688011960 con 0x7fd698105000 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.850+0000 7fd69f287700 1 -- 172.21.15.136:0/2403915697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 msgr2=0x7fd698105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.850+0000 7fd69f287700 1 --2- 172.21.15.136:0/2403915697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698105420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd6880098a0 tx=0x7fd688009bb0 comp rx=0 tx=0).stop 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.850+0000 7fd69f287700 1 -- 172.21.15.136:0/2403915697 shutdown_connections 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.850+0000 7fd69f287700 1 --2- 172.21.15.136:0/2403915697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698105420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.850+0000 7fd69f287700 1 -- 172.21.15.136:0/2403915697 >> 172.21.15.136:0/2403915697 conn(0x7fd6981005a0 msgr2=0x7fd6981029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.850+0000 7fd69f287700 1 -- 172.21.15.136:0/2403915697 shutdown_connections 2024-03-26T05:17:00.972 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.850+0000 7fd69f287700 1 -- 172.21.15.136:0/2403915697 wait complete. 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.851+0000 7fd69f287700 1 Processor -- start 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.851+0000 7fd69f287700 1 -- start start 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.851+0000 7fd69f287700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.851+0000 7fd69f287700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd698199330 con 0x7fd698105000 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.851+0000 7fd69d023700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.851+0000 7fd69d023700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39738/0 (socket says 172.21.15.136:39738) 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.851+0000 7fd69d023700 1 -- 172.21.15.136:0/2540276369 learned_addr learned my addr 172.21.15.136:0/2540276369 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.852+0000 7fd69d023700 1 -- 172.21.15.136:0/2540276369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688009550 con 0x7fd698105000 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.852+0000 7fd69d023700 1 --2- 172.21.15.136:0/2540276369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698198df0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd688006af0 tx=0x7fd6880119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.852+0000 7fd68e7fc700 1 -- 172.21.15.136:0/2540276369 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6880039b0 con 0x7fd698105000 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.852+0000 7fd68e7fc700 1 -- 172.21.15.136:0/2540276369 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fd688003b10 con 0x7fd698105000 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.852+0000 7fd68e7fc700 1 -- 172.21.15.136:0/2540276369 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd688019670 con 0x7fd698105000 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.852+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd698199530 con 0x7fd698105000 2024-03-26T05:17:00.973 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.852+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6981999d0 con 0x7fd698105000 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.853+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd698109430 con 0x7fd698105000 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.853+0000 7fd68e7fc700 1 -- 172.21.15.136:0/2540276369 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 12) v1 ==== 45906+0+0 (secure 0 0 0) 0x7fd6880198e0 con 0x7fd698105000 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.853+0000 7fd68e7fc700 1 --2- 172.21.15.136:0/2540276369 >> v2:172.21.15.136:6800/2365933975 conn(0x7fd684038ef0 0x7fd68403b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.853+0000 7fd68e7fc700 1 -- 172.21.15.136:0/2540276369 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fd68804d760 con 0x7fd698105000 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.853+0000 7fd69c822700 1 --2- 172.21.15.136:0/2540276369 >> v2:172.21.15.136:6800/2365933975 conn(0x7fd684038ef0 0x7fd68403b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.854+0000 7fd69c822700 1 --2- 172.21.15.136:0/2540276369 >> v2:172.21.15.136:6800/2365933975 conn(0x7fd684038ef0 0x7fd68403b3b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd694006fd0 tx=0x7fd694006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:16:59.858+0000 7fd68e7fc700 1 -- 172.21.15.136:0/2540276369 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd688023e00 con 0x7fd698105000 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.083+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd6980634d0 con 0x7fd684038ef0 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.311+0000 7fd68e7fc700 1 -- 172.21.15.136:0/2540276369 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fd6980634d0 con 0x7fd684038ef0 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 >> v2:172.21.15.136:6800/2365933975 conn(0x7fd684038ef0 msgr2=0x7fd68403b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 --2- 172.21.15.136:0/2540276369 >> v2:172.21.15.136:6800/2365933975 conn(0x7fd684038ef0 0x7fd68403b3b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd694006fd0 tx=0x7fd694006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 msgr2=0x7fd698198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:00.974 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 --2- 172.21.15.136:0/2540276369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698198df0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd688006af0 tx=0x7fd6880119e0 comp rx=0 tx=0).stop 2024-03-26T05:17:00.975 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 shutdown_connections 2024-03-26T05:17:00.975 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 --2- 172.21.15.136:0/2540276369 >> v2:172.21.15.136:6800/2365933975 conn(0x7fd684038ef0 0x7fd68403b3b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:00.975 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 --2- 172.21.15.136:0/2540276369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd698105000 0x7fd698198df0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:00.975 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 >> 172.21.15.136:0/2540276369 conn(0x7fd6981005a0 msgr2=0x7fd698101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:00.975 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 shutdown_connections 2024-03-26T05:17:00.975 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:00.313+0000 7fd69f287700 1 -- 172.21.15.136:0/2540276369 wait complete. 2024-03-26T05:17:00.975 INFO:teuthology.orchestra.run.smithi136.stdout:Fetching dashboard port number... 2024-03-26T05:17:01.751 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:01 smithi136 ceph-mon[29560]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:01.752 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:02.380 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T05:17:02.380 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.521+0000 7f8eb2a2c700 1 Processor -- start 2024-03-26T05:17:02.380 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.521+0000 7f8eb2a2c700 1 -- start start 2024-03-26T05:17:02.380 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eb2a2c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac108fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eb2a2c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac079f70 con 0x7f8eac106bc0 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eabfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac108fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eabfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac108fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44472/0 (socket says 172.21.15.136:44472) 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eabfff700 1 -- 172.21.15.136:0/2332632954 learned_addr learned my addr 172.21.15.136:0/2332632954 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eabfff700 1 -- 172.21.15.136:0/2332632954 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac1094f0 con 0x7f8eac106bc0 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eabfff700 1 --2- 172.21.15.136:0/2332632954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac108fb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0098a0 tx=0x7f8e9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4647f52124e634 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eaaffd700 1 -- 172.21.15.136:0/2332632954 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e9c004160 con 0x7f8eac106bc0 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eaaffd700 1 -- 172.21.15.136:0/2332632954 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f8e9c0042c0 con 0x7f8eac106bc0 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.522+0000 7f8eaaffd700 1 -- 172.21.15.136:0/2332632954 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e9c011960 con 0x7f8eac106bc0 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.523+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/2332632954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 msgr2=0x7f8eac108fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.523+0000 7f8eb2a2c700 1 --2- 172.21.15.136:0/2332632954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac108fb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0098a0 tx=0x7f8e9c009bb0 comp rx=0 tx=0).stop 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.523+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/2332632954 shutdown_connections 2024-03-26T05:17:02.381 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.523+0000 7f8eb2a2c700 1 --2- 172.21.15.136:0/2332632954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac108fb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.523+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/2332632954 >> 172.21.15.136:0/2332632954 conn(0x7f8eac1005d0 msgr2=0x7f8eac1029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.523+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/2332632954 shutdown_connections 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.523+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/2332632954 wait complete. 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.524+0000 7f8eb2a2c700 1 Processor -- start 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.524+0000 7f8eb2a2c700 1 -- start start 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.524+0000 7f8eb2a2c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac106440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.524+0000 7f8eb2a2c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac106980 con 0x7f8eac106bc0 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8eabfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac106440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8eabfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac106440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44486/0 (socket says 172.21.15.136:44486) 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8eabfff700 1 -- 172.21.15.136:0/507429319 learned_addr learned my addr 172.21.15.136:0/507429319 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:02.382 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8eabfff700 1 -- 172.21.15.136:0/507429319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c009550 con 0x7f8eac106bc0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8eabfff700 1 --2- 172.21.15.136:0/507429319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac106440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c004290 tx=0x7f8e9c011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8ea97fa700 1 -- 172.21.15.136:0/507429319 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e9c0036a0 con 0x7f8eac106bc0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac104b70 con 0x7f8eac106bc0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac105010 con 0x7f8eac106bc0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8ea97fa700 1 -- 172.21.15.136:0/507429319 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f8e9c024440 con 0x7f8eac106bc0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.525+0000 7f8ea97fa700 1 -- 172.21.15.136:0/507429319 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e9c01a560 con 0x7f8eac106bc0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.526+0000 7f8ea97fa700 1 -- 172.21.15.136:0/507429319 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 12) v1 ==== 45906+0+0 (secure 0 0 0) 0x7f8e9c01a6c0 con 0x7f8eac106bc0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.526+0000 7f8ea97fa700 1 --2- 172.21.15.136:0/507429319 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8e94038ef0 0x7f8e9403b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.526+0000 7f8ea97fa700 1 -- 172.21.15.136:0/507429319 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f8e9c04de10 con 0x7f8eac106bc0 2024-03-26T05:17:02.383 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.526+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e98005150 con 0x7f8eac106bc0 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.527+0000 7f8eab7fe700 1 --2- 172.21.15.136:0/507429319 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8e94038ef0 0x7f8e9403b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.527+0000 7f8eab7fe700 1 --2- 172.21.15.136:0/507429319 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8e94038ef0 0x7f8e9403b3b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0006fd0 tx=0x7f8ea0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.532+0000 7f8ea97fa700 1 -- 172.21.15.136:0/507429319 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8e9c02d910 con 0x7f8eac106bc0 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.747+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f8e98005980 con 0x7f8eac106bc0 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.748+0000 7f8ea97fa700 1 -- 172.21.15.136:0/507429319 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f8e9c018be0 con 0x7f8eac106bc0 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8e94038ef0 msgr2=0x7f8e9403b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 --2- 172.21.15.136:0/507429319 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8e94038ef0 0x7f8e9403b3b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0006fd0 tx=0x7f8ea0006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 msgr2=0x7f8eac106440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 --2- 172.21.15.136:0/507429319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac106440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c004290 tx=0x7f8e9c011eb0 comp rx=0 tx=0).stop 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 shutdown_connections 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 --2- 172.21.15.136:0/507429319 >> v2:172.21.15.136:6800/2365933975 conn(0x7f8e94038ef0 0x7f8e9403b3b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 --2- 172.21.15.136:0/507429319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eac106bc0 0x7f8eac106440 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:02.384 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 >> 172.21.15.136:0/507429319 conn(0x7f8eac1005d0 msgr2=0x7f8eac101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:02.385 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 shutdown_connections 2024-03-26T05:17:02.385 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:01.750+0000 7f8eb2a2c700 1 -- 172.21.15.136:0/507429319 wait complete. 2024-03-26T05:17:02.385 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T05:17:02.385 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-03-26T05:17:02.392 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T05:17:02.392 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-03-26T05:17:02.392 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-03-26T05:17:02.392 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T05:17:02.394 INFO:teuthology.orchestra.run.smithi136.stdout:Ceph Dashboard is now available at: 2024-03-26T05:17:02.394 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:02.394 INFO:teuthology.orchestra.run.smithi136.stdout: URL: https://smithi136.front.sepia.ceph.com:8443/ 2024-03-26T05:17:02.394 INFO:teuthology.orchestra.run.smithi136.stdout: User: admin 2024-03-26T05:17:02.394 INFO:teuthology.orchestra.run.smithi136.stdout: Password: rxibi7vss6 2024-03-26T05:17:02.394 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:02.395 INFO:teuthology.orchestra.run.smithi136.stdout:Saving cluster configuration to /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config directory 2024-03-26T05:17:02.395 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling autotune for osd_memory_target 2024-03-26T05:17:02.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/507429319' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T05:17:03.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:03 smithi136 ceph-mon[29560]: mgrmap e13: y(active, since 6s) 2024-03-26T05:17:04.249 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.825+0000 7f5bbdb5e700 1 Processor -- start 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.825+0000 7f5bbdb5e700 1 -- start start 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bbdb5e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8106520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bbdb5e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb8106af0 con 0x7f5bb8106100 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bb77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bb77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44492/0 (socket says 172.21.15.136:44492) 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bb77fe700 1 -- 172.21.15.136:0/3905990004 learned_addr learned my addr 172.21.15.136:0/3905990004 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bb77fe700 1 -- 172.21.15.136:0/3905990004 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb8107300 con 0x7f5bb8106100 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bb77fe700 1 --2- 172.21.15.136:0/3905990004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8106520 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80098a0 tx=0x7f5ba8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566c46ff63a2a39a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bb67fc700 1 -- 172.21.15.136:0/3905990004 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ba8004160 con 0x7f5bb8106100 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bb67fc700 1 -- 172.21.15.136:0/3905990004 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5ba80042c0 con 0x7f5bb8106100 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.826+0000 7f5bb67fc700 1 -- 172.21.15.136:0/3905990004 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ba8011960 con 0x7f5bb8106100 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.827+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/3905990004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 msgr2=0x7f5bb8106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:04.250 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.827+0000 7f5bbdb5e700 1 --2- 172.21.15.136:0/3905990004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8106520 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80098a0 tx=0x7f5ba8009bb0 comp rx=0 tx=0).stop 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.827+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/3905990004 shutdown_connections 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.827+0000 7f5bbdb5e700 1 --2- 172.21.15.136:0/3905990004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8106520 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.827+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/3905990004 >> 172.21.15.136:0/3905990004 conn(0x7f5bb81016a0 msgr2=0x7f5bb8103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.827+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/3905990004 shutdown_connections 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.827+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/3905990004 wait complete. 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bbdb5e700 1 Processor -- start 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bbdb5e700 1 -- start start 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bbdb5e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bbdb5e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb819a430 con 0x7f5bb8106100 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bb77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:04.251 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bb77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44498/0 (socket says 172.21.15.136:44498) 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bb77fe700 1 -- 172.21.15.136:0/2949169296 learned_addr learned my addr 172.21.15.136:0/2949169296 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bb77fe700 1 -- 172.21.15.136:0/2949169296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba8009550 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.828+0000 7f5bb77fe700 1 --2- 172.21.15.136:0/2949169296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8199ef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8006af0 tx=0x7f5ba80119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.829+0000 7f5bb4ff9700 1 -- 172.21.15.136:0/2949169296 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ba80039b0 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.829+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb819a630 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.829+0000 7f5bb4ff9700 1 -- 172.21.15.136:0/2949169296 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5ba8003b10 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.829+0000 7f5bb4ff9700 1 -- 172.21.15.136:0/2949169296 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ba8019670 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.829+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb819aad0 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.829+0000 7f5bb4ff9700 1 -- 172.21.15.136:0/2949169296 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f5ba8019960 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.829+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb810a530 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.830+0000 7f5bb4ff9700 1 --2- 172.21.15.136:0/2949169296 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5ba4038f90 0x7f5ba403b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.830+0000 7f5bb4ff9700 1 -- 172.21.15.136:0/2949169296 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f5ba804d970 con 0x7f5bb8106100 2024-03-26T05:17:04.252 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.830+0000 7f5bb6ffd700 1 --2- 172.21.15.136:0/2949169296 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5ba4038f90 0x7f5ba403b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.830+0000 7f5bb6ffd700 1 --2- 172.21.15.136:0/2949169296 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5ba4038f90 0x7f5ba403b450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0006fd0 tx=0x7f5ba0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:02.835+0000 7f5bb4ff9700 1 -- 172.21.15.136:0/2949169296 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5ba8018710 con 0x7f5bb8106100 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.042+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f5bb819d490 con 0x7f5bb8106100 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.042+0000 7f5bb4ff9700 1 -- 172.21.15.136:0/2949169296 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5ba8018350 con 0x7f5bb8106100 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5ba4038f90 msgr2=0x7f5ba403b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 --2- 172.21.15.136:0/2949169296 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5ba4038f90 0x7f5ba403b450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0006fd0 tx=0x7f5ba0006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 msgr2=0x7f5bb8199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 --2- 172.21.15.136:0/2949169296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8199ef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8006af0 tx=0x7f5ba80119e0 comp rx=0 tx=0).stop 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 shutdown_connections 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 --2- 172.21.15.136:0/2949169296 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5ba4038f90 0x7f5ba403b450 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 --2- 172.21.15.136:0/2949169296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb8106100 0x7f5bb8199ef0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 >> 172.21.15.136:0/2949169296 conn(0x7f5bb81016a0 msgr2=0x7f5bb8102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 shutdown_connections 2024-03-26T05:17:04.253 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:03.045+0000 7f5bbdb5e700 1 -- 172.21.15.136:0/2949169296 wait complete. 2024-03-26T05:17:05.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.712+0000 7fcf9b60c700 1 Processor -- start 2024-03-26T05:17:05.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.712+0000 7fcf9b60c700 1 -- start start 2024-03-26T05:17:05.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.712+0000 7fcf9b60c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9407a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:05.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.712+0000 7fcf9b60c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9407a9d0 con 0x7fcf9407bfa0 2024-03-26T05:17:05.632 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.712+0000 7fcf993a8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9407a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.712+0000 7fcf993a8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9407a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44502/0 (socket says 172.21.15.136:44502) 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.712+0000 7fcf993a8700 1 -- 172.21.15.136:0/2582847122 learned_addr learned my addr 172.21.15.136:0/2582847122 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.712+0000 7fcf993a8700 1 -- 172.21.15.136:0/2582847122 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9407ab10 con 0x7fcf9407bfa0 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf993a8700 1 --2- 172.21.15.136:0/2582847122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9407a400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098a0 tx=0x7fcf90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af74d9f88353ec7f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf8bfff700 1 -- 172.21.15.136:0/2582847122 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf90004160 con 0x7fcf9407bfa0 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf8bfff700 1 -- 172.21.15.136:0/2582847122 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fcf900042c0 con 0x7fcf9407bfa0 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf8bfff700 1 -- 172.21.15.136:0/2582847122 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf90011960 con 0x7fcf9407bfa0 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2582847122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 msgr2=0x7fcf9407a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf9b60c700 1 --2- 172.21.15.136:0/2582847122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9407a400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098a0 tx=0x7fcf90009bb0 comp rx=0 tx=0).stop 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2582847122 shutdown_connections 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf9b60c700 1 --2- 172.21.15.136:0/2582847122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9407a400 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2582847122 >> 172.21.15.136:0/2582847122 conn(0x7fcf941016b0 msgr2=0x7fcf94103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2582847122 shutdown_connections 2024-03-26T05:17:05.633 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.713+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2582847122 wait complete. 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.714+0000 7fcf9b60c700 1 Processor -- start 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.714+0000 7fcf9b60c700 1 -- start start 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.714+0000 7fcf9b60c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9419c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.714+0000 7fcf9b60c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9419c5d0 con 0x7fcf9407bfa0 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.714+0000 7fcf993a8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9419c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.714+0000 7fcf993a8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9419c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44514/0 (socket says 172.21.15.136:44514) 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.714+0000 7fcf993a8700 1 -- 172.21.15.136:0/2669904310 learned_addr learned my addr 172.21.15.136:0/2669904310 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.714+0000 7fcf993a8700 1 -- 172.21.15.136:0/2669904310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90009550 con 0x7fcf9407bfa0 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.715+0000 7fcf993a8700 1 --2- 172.21.15.136:0/2669904310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9419c090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf90006af0 tx=0x7fcf900119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.715+0000 7fcf8a7fc700 1 -- 172.21.15.136:0/2669904310 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf900039b0 con 0x7fcf9407bfa0 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.715+0000 7fcf8a7fc700 1 -- 172.21.15.136:0/2669904310 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fcf90003b10 con 0x7fcf9407bfa0 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.715+0000 7fcf8a7fc700 1 -- 172.21.15.136:0/2669904310 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf90019670 con 0x7fcf9407bfa0 2024-03-26T05:17:05.634 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.715+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9419c7d0 con 0x7fcf9407bfa0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.715+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9419cc70 con 0x7fcf9407bfa0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.716+0000 7fcf8a7fc700 1 -- 172.21.15.136:0/2669904310 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7fcf9001a430 con 0x7fcf9407bfa0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.716+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9410c510 con 0x7fcf9407bfa0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.716+0000 7fcf8a7fc700 1 --2- 172.21.15.136:0/2669904310 >> v2:172.21.15.136:6800/2365933975 conn(0x7fcf80038f90 0x7fcf8003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.716+0000 7fcf8a7fc700 1 -- 172.21.15.136:0/2669904310 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fcf9004d670 con 0x7fcf9407bfa0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.716+0000 7fcf98ba7700 1 --2- 172.21.15.136:0/2669904310 >> v2:172.21.15.136:6800/2365933975 conn(0x7fcf80038f90 0x7fcf8003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.716+0000 7fcf98ba7700 1 --2- 172.21.15.136:0/2669904310 >> v2:172.21.15.136:6800/2365933975 conn(0x7fcf80038f90 0x7fcf8003b450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcf84006fd0 tx=0x7fcf84006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:04.721+0000 7fcf8a7fc700 1 -- 172.21.15.136:0/2669904310 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcf90023390 con 0x7fcf9407bfa0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.022+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fcf9419f550 con 0x7fcf9407bfa0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.025+0000 7fcf8a7fc700 1 -- 172.21.15.136:0/2669904310 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fcf90023390 con 0x7fcf9407bfa0 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 >> v2:172.21.15.136:6800/2365933975 conn(0x7fcf80038f90 msgr2=0x7fcf8003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 --2- 172.21.15.136:0/2669904310 >> v2:172.21.15.136:6800/2365933975 conn(0x7fcf80038f90 0x7fcf8003b450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcf84006fd0 tx=0x7fcf84006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 msgr2=0x7fcf9419c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:05.635 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 --2- 172.21.15.136:0/2669904310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9419c090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf90006af0 tx=0x7fcf900119e0 comp rx=0 tx=0).stop 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 shutdown_connections 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 --2- 172.21.15.136:0/2669904310 >> v2:172.21.15.136:6800/2365933975 conn(0x7fcf80038f90 0x7fcf8003b450 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 --2- 172.21.15.136:0/2669904310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf9407bfa0 0x7fcf9419c090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 >> 172.21.15.136:0/2669904310 conn(0x7fcf941016b0 msgr2=0x7fcf94102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 shutdown_connections 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr 2024-03-26T05:17:05.027+0000 7fcf9b60c700 1 -- 172.21.15.136:0/2669904310 wait complete. 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: sudo /sbin/cephadm shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: sudo /sbin/cephadm shell 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:05.636 INFO:teuthology.orchestra.run.smithi136.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T05:17:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: ceph telemetry on 2024-03-26T05:17:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:05.637 INFO:teuthology.orchestra.run.smithi136.stdout:For more information see: 2024-03-26T05:17:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T05:17:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:05.637 INFO:teuthology.orchestra.run.smithi136.stdout:Bootstrap complete. 2024-03-26T05:17:05.673 INFO:tasks.cephadm:Fetching config... 2024-03-26T05:17:05.673 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:17:05.673 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T05:17:05.688 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T05:17:05.688 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:17:05.688 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T05:17:05.743 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T05:17:05.743 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:17:05.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T05:17:05.810 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T05:17:05.810 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:17:05.810 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T05:17:05.864 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T05:17:05.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNIcTXqe5lI6r5KMXd2SWzQrtf4xWNFn3AqIw6bFsV/h70MymjMjsOBL3TlJXteeCN1DkYmQNK9xluzcbBK073cc0pimIZMAi0wynxUfHiKHAKpFvNT/DT5AnmMq3eN5jwXIvkgH5IiqldelRrLLforTdp4scQYoD5AuS2RD+CS0m+3A9F69pp4JAp8vw9Z2NS1IvkqE3hTdNjQwThL7lqVfuBZ9W7gaaNlgl1WxA+eN0zEKD5+J42SX+W2ZEFQN5WGo32Ks8hifMifncajPJuhYKyEKEAQR+0dmRsOMj7OjvOlj3OC3dLpbG3Sqg6IOWsdgbKN3MpRORNIAdzMQrwPx+0xcZWt41pcrX7EUBIlidwg+JvVpGoi3Op67Vfy/6xM2AuiLLbXPhW9ij25I1VWOhMDzkQJM57Y97+9KUZC/HNOLkz5gqkuYDLEl2901l/u259K77fBTMlAbyDg0cSg07oo+KRQfE5fHnaiQeG4Xx2c1x64ihGmje6rYvQNc= ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T05:17:05.943 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNIcTXqe5lI6r5KMXd2SWzQrtf4xWNFn3AqIw6bFsV/h70MymjMjsOBL3TlJXteeCN1DkYmQNK9xluzcbBK073cc0pimIZMAi0wynxUfHiKHAKpFvNT/DT5AnmMq3eN5jwXIvkgH5IiqldelRrLLforTdp4scQYoD5AuS2RD+CS0m+3A9F69pp4JAp8vw9Z2NS1IvkqE3hTdNjQwThL7lqVfuBZ9W7gaaNlgl1WxA+eN0zEKD5+J42SX+W2ZEFQN5WGo32Ks8hifMifncajPJuhYKyEKEAQR+0dmRsOMj7OjvOlj3OC3dLpbG3Sqg6IOWsdgbKN3MpRORNIAdzMQrwPx+0xcZWt41pcrX7EUBIlidwg+JvVpGoi3Op67Vfy/6xM2AuiLLbXPhW9ij25I1VWOhMDzkQJM57Y97+9KUZC/HNOLkz5gqkuYDLEl2901l/u259K77fBTMlAbyDg0cSg07oo+KRQfE5fHnaiQeG4Xx2c1x64ihGmje6rYvQNc= ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:17:05.957 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNIcTXqe5lI6r5KMXd2SWzQrtf4xWNFn3AqIw6bFsV/h70MymjMjsOBL3TlJXteeCN1DkYmQNK9xluzcbBK073cc0pimIZMAi0wynxUfHiKHAKpFvNT/DT5AnmMq3eN5jwXIvkgH5IiqldelRrLLforTdp4scQYoD5AuS2RD+CS0m+3A9F69pp4JAp8vw9Z2NS1IvkqE3hTdNjQwThL7lqVfuBZ9W7gaaNlgl1WxA+eN0zEKD5+J42SX+W2ZEFQN5WGo32Ks8hifMifncajPJuhYKyEKEAQR+0dmRsOMj7OjvOlj3OC3dLpbG3Sqg6IOWsdgbKN3MpRORNIAdzMQrwPx+0xcZWt41pcrX7EUBIlidwg+JvVpGoi3Op67Vfy/6xM2AuiLLbXPhW9ij25I1VWOhMDzkQJM57Y97+9KUZC/HNOLkz5gqkuYDLEl2901l/u259K77fBTMlAbyDg0cSg07oo+KRQfE5fHnaiQeG4Xx2c1x64ihGmje6rYvQNc= ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T05:17:06.002 INFO:teuthology.orchestra.run.smithi164.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNIcTXqe5lI6r5KMXd2SWzQrtf4xWNFn3AqIw6bFsV/h70MymjMjsOBL3TlJXteeCN1DkYmQNK9xluzcbBK073cc0pimIZMAi0wynxUfHiKHAKpFvNT/DT5AnmMq3eN5jwXIvkgH5IiqldelRrLLforTdp4scQYoD5AuS2RD+CS0m+3A9F69pp4JAp8vw9Z2NS1IvkqE3hTdNjQwThL7lqVfuBZ9W7gaaNlgl1WxA+eN0zEKD5+J42SX+W2ZEFQN5WGo32Ks8hifMifncajPJuhYKyEKEAQR+0dmRsOMj7OjvOlj3OC3dLpbG3Sqg6IOWsdgbKN3MpRORNIAdzMQrwPx+0xcZWt41pcrX7EUBIlidwg+JvVpGoi3Op67Vfy/6xM2AuiLLbXPhW9ij25I1VWOhMDzkQJM57Y97+9KUZC/HNOLkz5gqkuYDLEl2901l/u259K77fBTMlAbyDg0cSg07oo+KRQfE5fHnaiQeG4Xx2c1x64ihGmje6rYvQNc= ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:17:06.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T05:17:06.186 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2669904310' entity='client.admin' 2024-03-26T05:17:06.253 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:08.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 -- 172.21.15.136:0/3169908873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c101ac0 msgr2=0x7f810c101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:08.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 --2- 172.21.15.136:0/3169908873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c101ac0 0x7f810c101ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8100009910 tx=0x7f8100009c20 comp rx=0 tx=0).stop 2024-03-26T05:17:08.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 -- 172.21.15.136:0/3169908873 shutdown_connections 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 --2- 172.21.15.136:0/3169908873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c101ac0 0x7f810c101ee0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 -- 172.21.15.136:0/3169908873 >> 172.21.15.136:0/3169908873 conn(0x7f810c0fd060 msgr2=0x7f810c0ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 -- 172.21.15.136:0/3169908873 shutdown_connections 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 -- 172.21.15.136:0/3169908873 wait complete. 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 Processor -- start 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.532+0000 7f8112f59700 1 -- start start 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8112f59700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c1968f0 0x7f810c196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8112f59700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c197250 con 0x7f810c1968f0 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8110cf5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c1968f0 0x7f810c196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8110cf5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c1968f0 0x7f810c196d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44530/0 (socket says 172.21.15.136:44530) 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8110cf5700 1 -- 172.21.15.136:0/1834799021 learned_addr learned my addr 172.21.15.136:0/1834799021 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8110cf5700 1 -- 172.21.15.136:0/1834799021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000095f0 con 0x7f810c1968f0 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8110cf5700 1 --2- 172.21.15.136:0/1834799021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c1968f0 0x7f810c196d10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8100005160 tx=0x7f8100004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:08.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8109ffb700 1 -- 172.21.15.136:0/1834799021 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810001d070 con 0x7f810c1968f0 2024-03-26T05:17:08.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c197450 con 0x7f810c1968f0 2024-03-26T05:17:08.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8109ffb700 1 -- 172.21.15.136:0/1834799021 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f810000bc00 con 0x7f810c1968f0 2024-03-26T05:17:08.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.533+0000 7f8109ffb700 1 -- 172.21.15.136:0/1834799021 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81000038b0 con 0x7f810c1968f0 2024-03-26T05:17:08.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.534+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c199fc0 con 0x7f810c1968f0 2024-03-26T05:17:08.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.535+0000 7f8109ffb700 1 -- 172.21.15.136:0/1834799021 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f8100003a10 con 0x7f810c1968f0 2024-03-26T05:17:08.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.535+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c190930 con 0x7f810c1968f0 2024-03-26T05:17:08.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.535+0000 7f8109ffb700 1 --2- 172.21.15.136:0/1834799021 >> v2:172.21.15.136:6800/2365933975 conn(0x7f80fc038f40 0x7f80fc03b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:08.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.535+0000 7f8109ffb700 1 -- 172.21.15.136:0/1834799021 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f810004cd60 con 0x7f810c1968f0 2024-03-26T05:17:08.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.535+0000 7f810bfff700 1 --2- 172.21.15.136:0/1834799021 >> v2:172.21.15.136:6800/2365933975 conn(0x7f80fc038f40 0x7f80fc03b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:08.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.535+0000 7f810bfff700 1 --2- 172.21.15.136:0/1834799021 >> v2:172.21.15.136:6800/2365933975 conn(0x7f80fc038f40 0x7f80fc03b400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f80f8006fd0 tx=0x7f80f8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:08.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.540+0000 7f8109ffb700 1 -- 172.21.15.136:0/1834799021 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8100029360 con 0x7f810c1968f0 2024-03-26T05:17:08.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.739+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f810c107260 con 0x7f810c1968f0 2024-03-26T05:17:08.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.744+0000 7f8109ffb700 1 -- 172.21.15.136:0/1834799021 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8100026070 con 0x7f810c1968f0 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.746+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 >> v2:172.21.15.136:6800/2365933975 conn(0x7f80fc038f40 msgr2=0x7f80fc03b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.746+0000 7f8112f59700 1 --2- 172.21.15.136:0/1834799021 >> v2:172.21.15.136:6800/2365933975 conn(0x7f80fc038f40 0x7f80fc03b400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f80f8006fd0 tx=0x7f80f8006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.746+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c1968f0 msgr2=0x7f810c196d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.746+0000 7f8112f59700 1 --2- 172.21.15.136:0/1834799021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c1968f0 0x7f810c196d10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8100005160 tx=0x7f8100004f00 comp rx=0 tx=0).stop 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.746+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 shutdown_connections 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.746+0000 7f8112f59700 1 --2- 172.21.15.136:0/1834799021 >> v2:172.21.15.136:6800/2365933975 conn(0x7f80fc038f40 0x7f80fc03b400 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.747+0000 7f8112f59700 1 --2- 172.21.15.136:0/1834799021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c1968f0 0x7f810c196d10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.747+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 >> 172.21.15.136:0/1834799021 conn(0x7f810c0fd060 msgr2=0x7f810c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.747+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 shutdown_connections 2024-03-26T05:17:08.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:08.747+0000 7f8112f59700 1 -- 172.21.15.136:0/1834799021 wait complete. 2024-03-26T05:17:08.748 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:08 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:08.748 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:08 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:08.749 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:08 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:17:08.749 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:08 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:08.749 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:08 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:08.749 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:08 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:09.354 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T05:17:09.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T05:17:09.587 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:10.014 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1834799021' entity='client.admin' 2024-03-26T05:17:10.015 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:09 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:10.015 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:09 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:11.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.055+0000 7f2cb7689700 1 -- 172.21.15.136:0/2016495448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 msgr2=0x7f2cb0102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:11.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.055+0000 7f2cb7689700 1 --2- 172.21.15.136:0/2016495448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 0x7f2cb0102460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009910 tx=0x7f2cac009c20 comp rx=0 tx=0).stop 2024-03-26T05:17:11.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.055+0000 7f2cb7689700 1 -- 172.21.15.136:0/2016495448 shutdown_connections 2024-03-26T05:17:11.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.055+0000 7f2cb7689700 1 --2- 172.21.15.136:0/2016495448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 0x7f2cb0102460 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:11.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.055+0000 7f2cb7689700 1 -- 172.21.15.136:0/2016495448 >> 172.21.15.136:0/2016495448 conn(0x7f2cb00fb6e0 msgr2=0x7f2cb00fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.055+0000 7f2cb7689700 1 -- 172.21.15.136:0/2016495448 shutdown_connections 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.055+0000 7f2cb7689700 1 -- 172.21.15.136:0/2016495448 wait complete. 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.056+0000 7f2cb7689700 1 Processor -- start 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.056+0000 7f2cb7689700 1 -- start start 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.056+0000 7f2cb7689700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 0x7f2cb01005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.056+0000 7f2cb7689700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb01993d0 con 0x7f2cb0102040 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.056+0000 7f2cb5425700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 0x7f2cb01005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.056+0000 7f2cb5425700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 0x7f2cb01005e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39708/0 (socket says 172.21.15.136:39708) 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.056+0000 7f2cb5425700 1 -- 172.21.15.136:0/241341442 learned_addr learned my addr 172.21.15.136:0/241341442 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.056+0000 7f2cb5425700 1 -- 172.21.15.136:0/241341442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cac0095f0 con 0x7f2cb0102040 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.057+0000 7f2cb5425700 1 --2- 172.21.15.136:0/241341442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 0x7f2cb01005e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2cac0057d0 tx=0x7f2cac005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.057+0000 7f2ca67fc700 1 -- 172.21.15.136:0/241341442 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cac01d070 con 0x7f2cb0102040 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.057+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb0199510 con 0x7f2cb0102040 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.057+0000 7f2ca67fc700 1 -- 172.21.15.136:0/241341442 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f2cac022470 con 0x7f2cb0102040 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.057+0000 7f2ca67fc700 1 -- 172.21.15.136:0/241341442 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cac00f5f0 con 0x7f2cb0102040 2024-03-26T05:17:11.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.057+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb01998f0 con 0x7f2cb0102040 2024-03-26T05:17:11.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.058+0000 7f2ca67fc700 1 -- 172.21.15.136:0/241341442 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f2cac00f870 con 0x7f2cb0102040 2024-03-26T05:17:11.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.058+0000 7f2ca67fc700 1 --2- 172.21.15.136:0/241341442 >> v2:172.21.15.136:6800/2365933975 conn(0x7f2c9c038f40 0x7f2c9c03b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:11.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.058+0000 7f2ca67fc700 1 -- 172.21.15.136:0/241341442 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f2cac04d2b0 con 0x7f2cb0102040 2024-03-26T05:17:11.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.058+0000 7f2cb4c24700 1 --2- 172.21.15.136:0/241341442 >> v2:172.21.15.136:6800/2365933975 conn(0x7f2c9c038f40 0x7f2c9c03b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:11.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.058+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb018f5d0 con 0x7f2cb0102040 2024-03-26T05:17:11.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.058+0000 7f2cb4c24700 1 --2- 172.21.15.136:0/241341442 >> v2:172.21.15.136:6800/2365933975 conn(0x7f2c9c038f40 0x7f2c9c03b400 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0006fd0 tx=0x7f2ca0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:11.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.063+0000 7f2ca67fc700 1 -- 172.21.15.136:0/241341442 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2cac027070 con 0x7f2cb0102040 2024-03-26T05:17:11.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.261+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cb0106070 con 0x7f2c9c038f40 2024-03-26T05:17:11.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.264+0000 7f2ca67fc700 1 -- 172.21.15.136:0/241341442 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2cb0106070 con 0x7f2c9c038f40 2024-03-26T05:17:11.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.266+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 >> v2:172.21.15.136:6800/2365933975 conn(0x7f2c9c038f40 msgr2=0x7f2c9c03b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:11.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.266+0000 7f2cb7689700 1 --2- 172.21.15.136:0/241341442 >> v2:172.21.15.136:6800/2365933975 conn(0x7f2c9c038f40 0x7f2c9c03b400 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0006fd0 tx=0x7f2ca0006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:11.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.266+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 msgr2=0x7f2cb01005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:11.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.266+0000 7f2cb7689700 1 --2- 172.21.15.136:0/241341442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 0x7f2cb01005e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2cac0057d0 tx=0x7f2cac005800 comp rx=0 tx=0).stop 2024-03-26T05:17:11.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.267+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 shutdown_connections 2024-03-26T05:17:11.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.267+0000 7f2cb7689700 1 --2- 172.21.15.136:0/241341442 >> v2:172.21.15.136:6800/2365933975 conn(0x7f2c9c038f40 0x7f2c9c03b400 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:11.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.267+0000 7f2cb7689700 1 --2- 172.21.15.136:0/241341442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0102040 0x7f2cb01005e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:11.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.267+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 >> 172.21.15.136:0/241341442 conn(0x7f2cb00fb6e0 msgr2=0x7f2cb00fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:11.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.267+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 shutdown_connections 2024-03-26T05:17:11.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:11.267+0000 7f2cb7689700 1 -- 172.21.15.136:0/241341442 wait complete. 2024-03-26T05:17:12.103 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi164 2024-03-26T05:17:12.103 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:17:12.103 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.conf 2024-03-26T05:17:12.121 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:17:12.121 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:17:12.179 INFO:tasks.cephadm:Adding host smithi164 to orchestrator... 2024-03-26T05:17:12.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch host add smithi164 2024-03-26T05:17:12.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:12.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:12.410 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:13.517 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:13 smithi136 ceph-mon[29560]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:17:13.517 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:13 smithi136 ceph-mon[29560]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:17:14.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.049+0000 7fa2ec840700 1 -- 172.21.15.136:0/3647995789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 msgr2=0x7fa2e8106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:14.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.049+0000 7fa2ec840700 1 --2- 172.21.15.136:0/3647995789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 0x7fa2e8106d20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000b0a0 tx=0x7fa2e000b3b0 comp rx=0 tx=0).stop 2024-03-26T05:17:14.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.050+0000 7fa2ec840700 1 -- 172.21.15.136:0/3647995789 shutdown_connections 2024-03-26T05:17:14.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.050+0000 7fa2ec840700 1 --2- 172.21.15.136:0/3647995789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 0x7fa2e8106d20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:14.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.050+0000 7fa2ec840700 1 -- 172.21.15.136:0/3647995789 >> 172.21.15.136:0/3647995789 conn(0x7fa2e80fc470 msgr2=0x7fa2e80fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:14.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.050+0000 7fa2ec840700 1 -- 172.21.15.136:0/3647995789 shutdown_connections 2024-03-26T05:17:14.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.050+0000 7fa2ec840700 1 -- 172.21.15.136:0/3647995789 wait complete. 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2ec840700 1 Processor -- start 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2ec840700 1 -- start start 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2ec840700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 0x7fa2e81978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2ec840700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e806ae20 con 0x7fa2e8106940 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2e659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 0x7fa2e81978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2e659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 0x7fa2e81978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39724/0 (socket says 172.21.15.136:39724) 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2e659c700 1 -- 172.21.15.136:0/1006368558 learned_addr learned my addr 172.21.15.136:0/1006368558 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2e659c700 1 -- 172.21.15.136:0/1006368558 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e0009d00 con 0x7fa2e8106940 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2e659c700 1 --2- 172.21.15.136:0/1006368558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 0x7fa2e81978c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0015040 tx=0x7fa2e0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2d77fe700 1 -- 172.21.15.136:0/1006368558 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2e000e040 con 0x7fa2e8106940 2024-03-26T05:17:14.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e8197ec0 con 0x7fa2e8106940 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2d77fe700 1 -- 172.21.15.136:0/1006368558 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa2e001db10 con 0x7fa2e8106940 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.051+0000 7fa2d77fe700 1 -- 172.21.15.136:0/1006368558 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2e0004550 con 0x7fa2e8106940 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.052+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e8198340 con 0x7fa2e8106940 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.052+0000 7fa2d77fe700 1 -- 172.21.15.136:0/1006368558 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7fa2e0019070 con 0x7fa2e8106940 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.052+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e8104bf0 con 0x7fa2e8106940 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.052+0000 7fa2d77fe700 1 --2- 172.21.15.136:0/1006368558 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa2d0038f90 0x7fa2d003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.053+0000 7fa2d77fe700 1 -- 172.21.15.136:0/1006368558 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fa2e004d920 con 0x7fa2e8106940 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.053+0000 7fa2e5d9b700 1 --2- 172.21.15.136:0/1006368558 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa2d0038f90 0x7fa2d003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:14.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.053+0000 7fa2e5d9b700 1 --2- 172.21.15.136:0/1006368558 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa2d0038f90 0x7fa2d003b450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc006fd0 tx=0x7fa2dc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:14.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.057+0000 7fa2d77fe700 1 -- 172.21.15.136:0/1006368558 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa2e001a380 con 0x7fa2e8106940 2024-03-26T05:17:14.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:14.260+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2e806a4d0 con 0x7fa2d0038f90 2024-03-26T05:17:15.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:14 smithi136 ceph-mon[29560]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:16.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:15 smithi136 ceph-mon[29560]: Deploying cephadm binary to smithi164 2024-03-26T05:17:17.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:17 smithi136 ceph-mon[29560]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:18.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.722+0000 7fa2d77fe700 1 -- 172.21.15.136:0/1006368558 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa2e806a4d0 con 0x7fa2d0038f90 2024-03-26T05:17:18.713 INFO:teuthology.orchestra.run.smithi136.stdout:Added host 'smithi164' with addr '172.21.15.164' 2024-03-26T05:17:18.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa2d0038f90 msgr2=0x7fa2d003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:18.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 --2- 172.21.15.136:0/1006368558 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa2d0038f90 0x7fa2d003b450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc006fd0 tx=0x7fa2dc006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:18.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 msgr2=0x7fa2e81978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:18.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 --2- 172.21.15.136:0/1006368558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 0x7fa2e81978c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0015040 tx=0x7fa2e0003ce0 comp rx=0 tx=0).stop 2024-03-26T05:17:18.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 shutdown_connections 2024-03-26T05:17:18.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 --2- 172.21.15.136:0/1006368558 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa2d0038f90 0x7fa2d003b450 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:18.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 --2- 172.21.15.136:0/1006368558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e8106940 0x7fa2e81978c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:18.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 >> 172.21.15.136:0/1006368558 conn(0x7fa2e80fc470 msgr2=0x7fa2e8108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:18.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 shutdown_connections 2024-03-26T05:17:18.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:18.725+0000 7fa2ec840700 1 -- 172.21.15.136:0/1006368558 wait complete. 2024-03-26T05:17:19.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T05:17:19.593 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:19.715 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:19 smithi136 ceph-mon[29560]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:19.715 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:19.715 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:19.715 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:20.738 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:20 smithi136 ceph-mon[29560]: Added host smithi164 2024-03-26T05:17:20.739 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:20 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.250+0000 7f333d7bb700 1 -- 172.21.15.136:0/3220612531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 msgr2=0x7f3338106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.250+0000 7f333d7bb700 1 --2- 172.21.15.136:0/3220612531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 0x7f3338106d40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3328009910 tx=0x7f3328009c20 comp rx=0 tx=0).stop 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.250+0000 7f333d7bb700 1 -- 172.21.15.136:0/3220612531 shutdown_connections 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.250+0000 7f333d7bb700 1 --2- 172.21.15.136:0/3220612531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 0x7f3338106d40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.250+0000 7f333d7bb700 1 -- 172.21.15.136:0/3220612531 >> 172.21.15.136:0/3220612531 conn(0x7f33380fc490 msgr2=0x7f33380fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.250+0000 7f333d7bb700 1 -- 172.21.15.136:0/3220612531 shutdown_connections 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.250+0000 7f333d7bb700 1 -- 172.21.15.136:0/3220612531 wait complete. 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.251+0000 7f333d7bb700 1 Processor -- start 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.251+0000 7f333d7bb700 1 -- start start 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.251+0000 7f333d7bb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 0x7f333819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.251+0000 7f333d7bb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333819ccc0 con 0x7f3338106960 2024-03-26T05:17:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.251+0000 7f3336ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 0x7f333819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:21.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.251+0000 7f3336ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 0x7f333819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43616/0 (socket says 172.21.15.136:43616) 2024-03-26T05:17:21.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.251+0000 7f3336ffd700 1 -- 172.21.15.136:0/1230812921 learned_addr learned my addr 172.21.15.136:0/1230812921 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:21.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.252+0000 7f3336ffd700 1 -- 172.21.15.136:0/1230812921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33280095f0 con 0x7f3338106960 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.252+0000 7f3336ffd700 1 --2- 172.21.15.136:0/1230812921 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 0x7f333819c780 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f332800f690 tx=0x7f332800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.252+0000 7f331ffff700 1 -- 172.21.15.136:0/1230812921 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f332801d070 con 0x7f3338106960 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.252+0000 7f331ffff700 1 -- 172.21.15.136:0/1230812921 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3328004500 con 0x7f3338106960 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.252+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333819cf20 con 0x7f3338106960 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.252+0000 7f331ffff700 1 -- 172.21.15.136:0/1230812921 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3328003b30 con 0x7f3338106960 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.252+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333819d3a0 con 0x7f3338106960 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.253+0000 7f331ffff700 1 -- 172.21.15.136:0/1230812921 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f332802b440 con 0x7f3338106960 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.253+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33380636b0 con 0x7f3338106960 2024-03-26T05:17:21.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.253+0000 7f331ffff700 1 --2- 172.21.15.136:0/1230812921 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3320038f40 0x7f332003b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:21.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.253+0000 7f331ffff700 1 -- 172.21.15.136:0/1230812921 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f332804cc20 con 0x7f3338106960 2024-03-26T05:17:21.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.253+0000 7f33367fc700 1 --2- 172.21.15.136:0/1230812921 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3320038f40 0x7f332003b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:21.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.254+0000 7f33367fc700 1 --2- 172.21.15.136:0/1230812921 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3320038f40 0x7f332003b400 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f332c006fd0 tx=0x7f332c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:21.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.258+0000 7f331ffff700 1 -- 172.21.15.136:0/1230812921 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3328015720 con 0x7f3338106960 2024-03-26T05:17:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.461+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f333810a470 con 0x7f3320038f40 2024-03-26T05:17:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.463+0000 7f331ffff700 1 -- 172.21.15.136:0/1230812921 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f333810a470 con 0x7f3320038f40 2024-03-26T05:17:21.453 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:17:21.453 INFO:teuthology.orchestra.run.smithi136.stdout:[{"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}, {"addr": "172.21.15.164", "hostname": "smithi164", "labels": [], "status": ""}] 2024-03-26T05:17:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3320038f40 msgr2=0x7f332003b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 --2- 172.21.15.136:0/1230812921 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3320038f40 0x7f332003b400 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f332c006fd0 tx=0x7f332c006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 msgr2=0x7f333819c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 --2- 172.21.15.136:0/1230812921 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 0x7f333819c780 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f332800f690 tx=0x7f332800f770 comp rx=0 tx=0).stop 2024-03-26T05:17:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 shutdown_connections 2024-03-26T05:17:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 --2- 172.21.15.136:0/1230812921 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3320038f40 0x7f332003b400 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 --2- 172.21.15.136:0/1230812921 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3338106960 0x7f333819c780 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 >> 172.21.15.136:0/1230812921 conn(0x7f33380fc490 msgr2=0x7f3338108700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 shutdown_connections 2024-03-26T05:17:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:21.465+0000 7f333d7bb700 1 -- 172.21.15.136:0/1230812921 wait complete. 2024-03-26T05:17:21.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:21 smithi136 ceph-mon[29560]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:21.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:22.006 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T05:17:22.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T05:17:22.238 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:22.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:22 smithi136 ceph-mon[29560]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:17:23.730 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:23 smithi136 ceph-mon[29560]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:23.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.751+0000 7fe551b0f700 1 -- 172.21.15.136:0/1766364234 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 msgr2=0x7fe54c106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:23.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.751+0000 7fe551b0f700 1 --2- 172.21.15.136:0/1766364234 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 0x7fe54c106e70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00b0a0 tx=0x7fe53c00b3b0 comp rx=0 tx=0).stop 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.752+0000 7fe551b0f700 1 -- 172.21.15.136:0/1766364234 shutdown_connections 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.752+0000 7fe551b0f700 1 --2- 172.21.15.136:0/1766364234 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 0x7fe54c106e70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.752+0000 7fe551b0f700 1 -- 172.21.15.136:0/1766364234 >> 172.21.15.136:0/1766364234 conn(0x7fe54c077070 msgr2=0x7fe54c077480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.752+0000 7fe551b0f700 1 -- 172.21.15.136:0/1766364234 shutdown_connections 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.752+0000 7fe551b0f700 1 -- 172.21.15.136:0/1766364234 wait complete. 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.752+0000 7fe551b0f700 1 Processor -- start 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.752+0000 7fe551b0f700 1 -- start start 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe551b0f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 0x7fe54c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe551b0f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c19cc30 con 0x7fe54c106a90 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe54b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 0x7fe54c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe54b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 0x7fe54c19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43626/0 (socket says 172.21.15.136:43626) 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe54b7fe700 1 -- 172.21.15.136:0/1265511790 learned_addr learned my addr 172.21.15.136:0/1265511790 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe54b7fe700 1 -- 172.21.15.136:0/1265511790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c009d00 con 0x7fe54c106a90 2024-03-26T05:17:23.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe54b7fe700 1 --2- 172.21.15.136:0/1265511790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 0x7fe54c19c6f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe53c015040 tx=0x7fe53c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:23.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe548ff9700 1 -- 172.21.15.136:0/1265511790 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe53c00e040 con 0x7fe54c106a90 2024-03-26T05:17:23.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe548ff9700 1 -- 172.21.15.136:0/1265511790 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe53c01db10 con 0x7fe54c106a90 2024-03-26T05:17:23.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19ce30 con 0x7fe54c106a90 2024-03-26T05:17:23.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe548ff9700 1 -- 172.21.15.136:0/1265511790 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe53c004570 con 0x7fe54c106a90 2024-03-26T05:17:23.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.753+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c19d2b0 con 0x7fe54c106a90 2024-03-26T05:17:23.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.754+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c0636b0 con 0x7fe54c106a90 2024-03-26T05:17:23.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.754+0000 7fe548ff9700 1 -- 172.21.15.136:0/1265511790 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7fe53c019070 con 0x7fe54c106a90 2024-03-26T05:17:23.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.754+0000 7fe548ff9700 1 --2- 172.21.15.136:0/1265511790 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe534038f40 0x7fe53403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:23.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.754+0000 7fe548ff9700 1 -- 172.21.15.136:0/1265511790 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fe53c04da70 con 0x7fe54c106a90 2024-03-26T05:17:23.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.754+0000 7fe54affd700 1 --2- 172.21.15.136:0/1265511790 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe534038f40 0x7fe53403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:23.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.755+0000 7fe54affd700 1 --2- 172.21.15.136:0/1265511790 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe534038f40 0x7fe53403b400 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe540006fd0 tx=0x7fe540006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:23.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.759+0000 7fe548ff9700 1 -- 172.21.15.136:0/1265511790 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe53c01a380 con 0x7fe54c106a90 2024-03-26T05:17:23.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:23.960+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe54c10b690 con 0x7fe54c106a90 2024-03-26T05:17:24.510 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.520+0000 7fe548ff9700 1 -- 172.21.15.136:0/1265511790 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe53c021070 con 0x7fe54c106a90 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe534038f40 msgr2=0x7fe53403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 --2- 172.21.15.136:0/1265511790 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe534038f40 0x7fe53403b400 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe540006fd0 tx=0x7fe540006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 msgr2=0x7fe54c19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 --2- 172.21.15.136:0/1265511790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 0x7fe54c19c6f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe53c015040 tx=0x7fe53c003ce0 comp rx=0 tx=0).stop 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 shutdown_connections 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 --2- 172.21.15.136:0/1265511790 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe534038f40 0x7fe53403b400 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 --2- 172.21.15.136:0/1265511790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe54c106a90 0x7fe54c19c6f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 >> 172.21.15.136:0/1265511790 conn(0x7fe54c077070 msgr2=0x7fe54c107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 shutdown_connections 2024-03-26T05:17:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:24.523+0000 7fe551b0f700 1 -- 172.21.15.136:0/1265511790 wait complete. 2024-03-26T05:17:24.514 INFO:teuthology.orchestra.run.smithi136.stderr:adjusted tunables profile to default 2024-03-26T05:17:24.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1265511790' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T05:17:24.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:24.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:24.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:24.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:24.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:17:24.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:24.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:25.178 INFO:tasks.cephadm:Adding mon.a on smithi136 2024-03-26T05:17:25.178 INFO:tasks.cephadm:Adding mon.c on smithi136 2024-03-26T05:17:25.178 INFO:tasks.cephadm:Adding mon.b on smithi164 2024-03-26T05:17:25.178 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi164:172.21.15.164=b' 2024-03-26T05:17:25.412 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:25.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:25 smithi136 ceph-mon[29560]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T05:17:25.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:25 smithi136 ceph-mon[29560]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:25.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1265511790' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T05:17:25.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:25 smithi136 ceph-mon[29560]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T05:17:25.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:25 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:25.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:25 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:25.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:25 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:26.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:26 smithi136 ceph-mon[29560]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:26.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:26 smithi136 ceph-mon[29560]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:17:26.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:26 smithi136 ceph-mon[29560]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:17:26.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.865+0000 7f556d380700 1 -- 172.21.15.164:0/673182986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 msgr2=0x7f5568106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:26.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.865+0000 7f556d380700 1 --2- 172.21.15.164:0/673182986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 0x7f5568106d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5558009910 tx=0x7f5558009c20 comp rx=0 tx=0).stop 2024-03-26T05:17:26.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.865+0000 7f556d380700 1 -- 172.21.15.164:0/673182986 shutdown_connections 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.865+0000 7f556d380700 1 --2- 172.21.15.164:0/673182986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 0x7f5568106d40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.865+0000 7f556d380700 1 -- 172.21.15.164:0/673182986 >> 172.21.15.164:0/673182986 conn(0x7f55680fc490 msgr2=0x7f55680fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.865+0000 7f556d380700 1 -- 172.21.15.164:0/673182986 shutdown_connections 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.865+0000 7f556d380700 1 -- 172.21.15.164:0/673182986 wait complete. 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.865+0000 7f556d380700 1 Processor -- start 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.866+0000 7f556d380700 1 -- start start 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.866+0000 7f556d380700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 0x7f556819b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.866+0000 7f556d380700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556819bbc0 con 0x7f5568106960 2024-03-26T05:17:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.866+0000 7f5566ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 0x7f556819b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.866+0000 7f5566ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 0x7f556819b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.164:59914/0 (socket says 172.21.15.164:59914) 2024-03-26T05:17:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.866+0000 7f5566ffd700 1 -- 172.21.15.164:0/3267469178 learned_addr learned my addr 172.21.15.164:0/3267469178 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:17:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.866+0000 7f5566ffd700 1 -- 172.21.15.164:0/3267469178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55580095f0 con 0x7f5568106960 2024-03-26T05:17:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.867+0000 7f5566ffd700 1 --2- 172.21.15.164:0/3267469178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 0x7f556819b680 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f555801a890 tx=0x7f555801a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.867+0000 7f554ffff700 1 -- 172.21.15.164:0/3267469178 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f555801f070 con 0x7f5568106960 2024-03-26T05:17:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.867+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556819bdc0 con 0x7f5568106960 2024-03-26T05:17:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.867+0000 7f554ffff700 1 -- 172.21.15.164:0/3267469178 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5558004500 con 0x7f5568106960 2024-03-26T05:17:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.867+0000 7f554ffff700 1 -- 172.21.15.164:0/3267469178 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5558003d90 con 0x7f5568106960 2024-03-26T05:17:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.867+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556819c240 con 0x7f5568106960 2024-03-26T05:17:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.868+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55681020a0 con 0x7f5568106960 2024-03-26T05:17:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.868+0000 7f554ffff700 1 -- 172.21.15.164:0/3267469178 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f5558004050 con 0x7f5568106960 2024-03-26T05:17:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.868+0000 7f554ffff700 1 --2- 172.21.15.164:0/3267469178 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5550038f40 0x7f555003b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.868+0000 7f554ffff700 1 -- 172.21.15.164:0/3267469178 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f555804f140 con 0x7f5568106960 2024-03-26T05:17:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.868+0000 7f55667fc700 1 --2- 172.21.15.164:0/3267469178 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5550038f40 0x7f555003b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.869+0000 7f55667fc700 1 --2- 172.21.15.164:0/3267469178 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5550038f40 0x7f555003b400 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f555c006fd0 tx=0x7f555c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:26.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:26.873+0000 7f554ffff700 1 -- 172.21.15.164:0/3267469178 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5558004670 con 0x7f5568106960 2024-03-26T05:17:27.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.079+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi164:172.21.15.164=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f5568196e80 con 0x7f5550038f40 2024-03-26T05:17:27.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.085+0000 7f554ffff700 1 -- 172.21.15.164:0/3267469178 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5568196e80 con 0x7f5550038f40 2024-03-26T05:17:27.075 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled mon update... 2024-03-26T05:17:27.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5550038f40 msgr2=0x7f555003b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 --2- 172.21.15.164:0/3267469178 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5550038f40 0x7f555003b400 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f555c006fd0 tx=0x7f555c006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 msgr2=0x7f556819b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 --2- 172.21.15.164:0/3267469178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 0x7f556819b680 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f555801a890 tx=0x7f555801a970 comp rx=0 tx=0).stop 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 shutdown_connections 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 --2- 172.21.15.164:0/3267469178 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5550038f40 0x7f555003b400 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 --2- 172.21.15.164:0/3267469178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5568106960 0x7f556819b680 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 >> 172.21.15.164:0/3267469178 conn(0x7f55680fc490 msgr2=0x7f5568108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.087+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 shutdown_connections 2024-03-26T05:17:27.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:27.088+0000 7f556d380700 1 -- 172.21.15.164:0/3267469178 wait complete. 2024-03-26T05:17:27.752 DEBUG:teuthology.orchestra.run.smithi136:mon.c> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T05:17:27.755 DEBUG:teuthology.orchestra.run.smithi164:mon.b> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T05:17:27.758 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T05:17:27.758 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T05:17:27.781 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:27 smithi136 ceph-mon[29560]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:27.781 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:27.781 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:27.782 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:27.782 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:27.782 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:27.782 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:27.782 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:28.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:28 smithi136 ceph-mon[29560]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi164:172.21.15.164=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:28.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:28 smithi136 ceph-mon[29560]: Saving service mon spec with placement smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi164:172.21.15.164=b;count:3 2024-03-26T05:17:28.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:28 smithi136 ceph-mon[29560]: Deploying daemon mon.b on smithi164 2024-03-26T05:17:29.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:29 smithi136 ceph-mon[29560]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:30.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.868+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3440735719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 msgr2=0x7fe024103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.868+0000 7fe02a2e6700 1 --2- 172.21.15.164:0/3440735719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 0x7fe024103010 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe018009910 tx=0x7fe018009c20 comp rx=0 tx=0).stop 2024-03-26T05:17:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.868+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3440735719 shutdown_connections 2024-03-26T05:17:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.868+0000 7fe02a2e6700 1 --2- 172.21.15.164:0/3440735719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 0x7fe024103010 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.868+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3440735719 >> 172.21.15.164:0/3440735719 conn(0x7fe0240fc480 msgr2=0x7fe0240fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.868+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3440735719 shutdown_connections 2024-03-26T05:17:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.868+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3440735719 wait complete. 2024-03-26T05:17:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.869+0000 7fe02a2e6700 1 Processor -- start 2024-03-26T05:17:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.869+0000 7fe02a2e6700 1 -- start start 2024-03-26T05:17:30.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.869+0000 7fe02a2e6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 0x7fe02419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.869+0000 7fe02a2e6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02419cce0 con 0x7fe024102c30 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.869+0000 7fe023fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 0x7fe02419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.869+0000 7fe023fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 0x7fe02419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.164:59936/0 (socket says 172.21.15.164:59936) 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.869+0000 7fe023fff700 1 -- 172.21.15.164:0/3181731311 learned_addr learned my addr 172.21.15.164:0/3181731311 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.870+0000 7fe023fff700 1 -- 172.21.15.164:0/3181731311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180095f0 con 0x7fe024102c30 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.870+0000 7fe023fff700 1 --2- 172.21.15.164:0/3181731311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 0x7fe02419c7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe01800f690 tx=0x7fe01800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.870+0000 7fe0217fa700 1 -- 172.21.15.164:0/3181731311 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe01801d070 con 0x7fe024102c30 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.870+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02419cee0 con 0x7fe024102c30 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.870+0000 7fe0217fa700 1 -- 172.21.15.164:0/3181731311 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe018004500 con 0x7fe024102c30 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.870+0000 7fe0217fa700 1 -- 172.21.15.164:0/3181731311 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe018003c10 con 0x7fe024102c30 2024-03-26T05:17:30.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.870+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02419d300 con 0x7fe024102c30 2024-03-26T05:17:30.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.871+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0240636b0 con 0x7fe024102c30 2024-03-26T05:17:30.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.871+0000 7fe0217fa700 1 -- 172.21.15.164:0/3181731311 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7fe018004050 con 0x7fe024102c30 2024-03-26T05:17:30.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.871+0000 7fe0217fa700 1 --2- 172.21.15.164:0/3181731311 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe00c038f40 0x7fe00c03b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:30.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.871+0000 7fe0217fa700 1 -- 172.21.15.164:0/3181731311 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fe01804cd60 con 0x7fe024102c30 2024-03-26T05:17:30.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.872+0000 7fe0237fe700 1 --2- 172.21.15.164:0/3181731311 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe00c038f40 0x7fe00c03b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:30.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.872+0000 7fe0237fe700 1 --2- 172.21.15.164:0/3181731311 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe00c038f40 0x7fe00c03b400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe014006fd0 tx=0x7fe014006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:30.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:30.877+0000 7fe0217fa700 1 -- 172.21.15.164:0/3181731311 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe018015530 con 0x7fe024102c30 2024-03-26T05:17:31.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.155+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe02410b650 con 0x7fe024102c30 2024-03-26T05:17:31.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.156+0000 7fe0217fa700 1 -- 172.21.15.164:0/3181731311 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7fe018026070 con 0x7fe024102c30 2024-03-26T05:17:31.146 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:17:31.146 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":1,"fsid":"d69b39a8-eb2f-11ee-95d0-87774f69a715","modified":"2024-03-26T05:15:52.811312Z","created":"2024-03-26T05:15:52.811312Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe00c038f40 msgr2=0x7fe00c03b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 --2- 172.21.15.164:0/3181731311 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe00c038f40 0x7fe00c03b400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe014006fd0 tx=0x7fe014006e40 comp rx=0 tx=0).stop 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 msgr2=0x7fe02419c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 --2- 172.21.15.164:0/3181731311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 0x7fe02419c7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe01800f690 tx=0x7fe01800f770 comp rx=0 tx=0).stop 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 shutdown_connections 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 --2- 172.21.15.164:0/3181731311 >> v2:172.21.15.136:6800/2365933975 conn(0x7fe00c038f40 0x7fe00c03b400 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 --2- 172.21.15.164:0/3181731311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe024102c30 0x7fe02419c7a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 >> 172.21.15.164:0/3181731311 conn(0x7fe0240fc480 msgr2=0x7fe0240fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:31.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 shutdown_connections 2024-03-26T05:17:31.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:31.158+0000 7fe02a2e6700 1 -- 172.21.15.164:0/3181731311 wait complete. 2024-03-26T05:17:31.149 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 1 2024-03-26T05:17:31.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:31 smithi136 ceph-mon[29560]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/3181731311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:17:32.760 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:32 smithi164 ceph-mon[33554]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T05:17:32.975 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T05:17:32.975 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T05:17:33.208 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:17:33.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:32 smithi164 bash[33535]: 3ae84517558222e3d3b92d696d0fc75ec974b952c1b7f2ee877d5d284b14fbc3 2024-03-26T05:17:33.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:32 smithi164 systemd[1]: Started Ceph mon.b for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:17:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.875+0000 7f0f96e8f700 1 -- 172.21.15.164:0/1723352382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 msgr2=0x7f0f80005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.875+0000 7f0f96e8f700 1 --2- 172.21.15.164:0/1723352382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 0x7f0f80005a50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0f90101a80 tx=0x7f0f84013e90 comp rx=0 tx=0).stop 2024-03-26T05:17:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.876+0000 7f0f96e8f700 1 -- 172.21.15.164:0/1723352382 shutdown_connections 2024-03-26T05:17:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.876+0000 7f0f96e8f700 1 --2- 172.21.15.164:0/1723352382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 0x7f0f80005a50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.876+0000 7f0f96e8f700 1 -- 172.21.15.164:0/1723352382 >> 172.21.15.164:0/1723352382 conn(0x7f0f900fd620 msgr2=0x7f0f900ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.876+0000 7f0f96e8f700 1 -- 172.21.15.164:0/1723352382 shutdown_connections 2024-03-26T05:17:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.876+0000 7f0f96e8f700 1 -- 172.21.15.164:0/1723352382 wait complete. 2024-03-26T05:17:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.876+0000 7f0f96e8f700 1 Processor -- start 2024-03-26T05:17:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f96e8f700 1 -- start start 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f96e8f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 0x7f0f9019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f96e8f700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 0x7f0f90196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f96e8f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9019d390 con 0x7f0f800035c0 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f96e8f700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9019d500 con 0x7f0f9019ce00 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f8ffff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 0x7f0f90196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f8ffff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 0x7f0f90196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57668/0 (socket says 172.21.15.164:57668) 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f94c2b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 0x7f0f9019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f8ffff700 1 -- 172.21.15.164:0/2364557010 learned_addr learned my addr 172.21.15.164:0/2364557010 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f8ffff700 1 -- 172.21.15.164:0/2364557010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 msgr2=0x7f0f90196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f8ffff700 1 -- 172.21.15.164:0/2364557010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 msgr2=0x7f0f90196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f8ffff700 1 --2- 172.21.15.164:0/2364557010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 0x7f0f90196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T05:17:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f8ffff700 1 --2- 172.21.15.164:0/2364557010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 0x7f0f90196a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f94c2b700 1 -- 172.21.15.164:0/2364557010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 msgr2=0x7f0f90196a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f94c2b700 1 --2- 172.21.15.164:0/2364557010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 0x7f0f90196a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.877+0000 7f0f94c2b700 1 -- 172.21.15.164:0/2364557010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840138e0 con 0x7f0f800035c0 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.878+0000 7f0f94c2b700 1 --2- 172.21.15.164:0/2364557010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 0x7f0f9019c8c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0f8401c000 tx=0x7f0f8400ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.878+0000 7f0f8dffb700 1 -- 172.21.15.164:0/2364557010 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f0f84018070 con 0x7f0f800035c0 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.878+0000 7f0f8dffb700 1 -- 172.21.15.164:0/2364557010 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f0f84005470 con 0x7f0f800035c0 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.878+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f90197080 con 0x7f0f800035c0 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.878+0000 7f0f8dffb700 1 -- 172.21.15.164:0/2364557010 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f0f84004560 con 0x7f0f800035c0 2024-03-26T05:17:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.878+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f901975d0 con 0x7f0f800035c0 2024-03-26T05:17:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.879+0000 7f0f8dffb700 1 -- 172.21.15.164:0/2364557010 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f0f840046c0 con 0x7f0f800035c0 2024-03-26T05:17:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.879+0000 7f0f8dffb700 1 --2- 172.21.15.164:0/2364557010 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0f74038d30 0x7f0f7403b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.879+0000 7f0f8dffb700 1 -- 172.21.15.164:0/2364557010 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f0f8404f770 con 0x7f0f800035c0 2024-03-26T05:17:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.879+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f901909b0 con 0x7f0f800035c0 2024-03-26T05:17:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.879+0000 7f0f8ffff700 1 --2- 172.21.15.164:0/2364557010 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0f74038d30 0x7f0f7403b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.880+0000 7f0f8ffff700 1 --2- 172.21.15.164:0/2364557010 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0f74038d30 0x7f0f7403b1f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c009180 tx=0x7f0f7c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:37.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:37.885+0000 7f0f8dffb700 1 -- 172.21.15.164:0/2364557010 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0f8402d790 con 0x7f0f800035c0 2024-03-26T05:17:38.130 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: mon.a calling monitor election 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: mon.b calling monitor election 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T05:17:38.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: fsmap 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: mgrmap e13: y(active, since 41s) 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: overall HEALTH_OK 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:38.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:37 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.157+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f90197840 con 0x7f0f800035c0 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: mon.a calling monitor election 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: mon.b calling monitor election 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: fsmap 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: mgrmap e13: y(active, since 41s) 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: overall HEALTH_OK 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:38.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:37 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:38.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.158+0000 7f0f8dffb700 1 -- 172.21.15.164:0/2364557010 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7f0f840103d0 con 0x7f0f800035c0 2024-03-26T05:17:38.150 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:17:38.150 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":2,"fsid":"d69b39a8-eb2f-11ee-95d0-87774f69a715","modified":"2024-03-26T05:17:32.769159Z","created":"2024-03-26T05:15:52.811312Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0f74038d30 msgr2=0x7f0f7403b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 --2- 172.21.15.164:0/2364557010 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0f74038d30 0x7f0f7403b1f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c009180 tx=0x7f0f7c008040 comp rx=0 tx=0).stop 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 msgr2=0x7f0f9019c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 --2- 172.21.15.164:0/2364557010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 0x7f0f9019c8c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0f8401c000 tx=0x7f0f8400ff20 comp rx=0 tx=0).stop 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 shutdown_connections 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 --2- 172.21.15.164:0/2364557010 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0f74038d30 0x7f0f7403b1f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 --2- 172.21.15.164:0/2364557010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f800035c0 0x7f0f9019c8c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 --2- 172.21.15.164:0/2364557010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9019ce00 0x7f0f90196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 >> 172.21.15.164:0/2364557010 conn(0x7f0f900fd620 msgr2=0x7f0f900fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:38.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 shutdown_connections 2024-03-26T05:17:38.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:38.161+0000 7f0f96e8f700 1 -- 172.21.15.164:0/2364557010 wait complete. 2024-03-26T05:17:38.152 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 2 2024-03-26T05:17:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:38 smithi164 ceph-mon[33554]: Deploying daemon mon.c on smithi136 2024-03-26T05:17:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/2364557010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:17:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:38 smithi164 ceph-mon[33554]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:38 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:39.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:38 smithi136 ceph-mon[29560]: Deploying daemon mon.c on smithi136 2024-03-26T05:17:39.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/2364557010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:17:39.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:38 smithi136 ceph-mon[29560]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:39.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:38 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:39.741 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T05:17:39.742 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T05:17:39.982 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:17:41.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:40 smithi136 systemd[1]: Starting Ceph mon.c for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:17:41.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 podman[38087]: 2024-03-26 05:17:40.991425286 +0000 UTC m=+0.018333797 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:17:41.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 podman[38087]: 2024-03-26 05:17:41.100134555 +0000 UTC m=+0.127043057 container create 9a92e0eeb3f7ab6b9dd342808b0509df18b4c93dd538996d81f337c9a3b53c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T05:17:41.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 podman[38087]: 2024-03-26 05:17:41.31079766 +0000 UTC m=+0.337706163 container init 9a92e0eeb3f7ab6b9dd342808b0509df18b4c93dd538996d81f337c9a3b53c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True) 2024-03-26T05:17:41.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 podman[38087]: 2024-03-26 05:17:41.314468039 +0000 UTC m=+0.341376542 container start 9a92e0eeb3f7ab6b9dd342808b0509df18b4c93dd538996d81f337c9a3b53c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-c, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T05:17:41.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T05:17:41.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T05:17:41.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pidfile_write: ignore empty --pid-file 2024-03-26T05:17:41.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: load: jerasure load: lrc 2024-03-26T05:17:41.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: RocksDB version: 7.9.2 2024-03-26T05:17:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Git sha 0 2024-03-26T05:17:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T05:17:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: DB SUMMARY 2024-03-26T05:17:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: DB Session ID: Q1G6QCFCJA951B5S2N99 2024-03-26T05:17:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: CURRENT file: CURRENT 2024-03-26T05:17:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T05:17:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T05:17:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.error_if_exists: 0 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.create_if_missing: 0 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.paranoid_checks: 1 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.env: 0x563d03859ae0 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.info_log: 0x563d059b3920 2024-03-26T05:17:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.statistics: (nil) 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.use_fsync: 0 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_log_file_size: 0 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.allow_fallocate: 1 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.use_direct_reads: 0 2024-03-26T05:17:41.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.db_log_dir: 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.wal_dir: 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.write_buffer_manager: 0x563d059861e0 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T05:17:41.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.unordered_write: 0 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.row_cache: None 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.wal_filter: None 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.two_write_queues: 0 2024-03-26T05:17:41.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.wal_compression: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.atomic_flush: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.log_readahead_size: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T05:17:41.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_background_jobs: 2 2024-03-26T05:17:41.831 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_background_compactions: -1 2024-03-26T05:17:41.831 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_subcompactions: 1 2024-03-26T05:17:41.831 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T05:17:41.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T05:17:41.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T05:17:41.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T05:17:41.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T05:17:41.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T05:17:41.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_open_files: -1 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_background_flushes: -1 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Compression algorithms supported: 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: kZSTD supported: 0 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: kXpressCompression supported: 0 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: kZlibCompression supported: 1 2024-03-26T05:17:41.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: kSnappyCompression supported: 1 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: kLZ4Compression supported: 1 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: kBZip2Compression supported: 0 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.merge_operator: 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_filter: None 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T05:17:41.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563d059b3a20) 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_type: 0 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_index_type: 0 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_shortening: 1 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: checksum: 4 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: no_block_cache: 0 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache: 0x563d059b7090 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-03-26T05:17:41.835 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_options: 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: capacity : 536870912 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_shard_bits : 4 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: strict_capacity_limit : 0 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_compressed: (nil) 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: persistent_cache: (nil) 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size: 4096 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size_deviation: 10 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_restart_interval: 16 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_block_restart_interval: 1 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: metadata_block_size: 4096 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: partition_filters: 0 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: use_delta_encoding: 1 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: filter_policy: bloomfilter 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: whole_key_filtering: 1 2024-03-26T05:17:41.836 INFO:journalctl@ceph.mon.c.smithi136.stdout: verify_compression: 0 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout: format_version: 5 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_index_compression: 1 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_align: 0 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout: max_auto_readahead_size: 262144 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout: prepopulate_block_cache: 0 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout: initial_auto_readahead_size: 8192 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression: NoCompression 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.num_levels: 7 2024-03-26T05:17:41.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T05:17:41.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T05:17:41.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T05:17:41.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.table_properties_collectors: 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.inplace_update_support: 0 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T05:17:41.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.bloom_locality: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.max_successive_merges: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.ttl: 2592000 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.enable_blob_files: false 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.min_blob_size: 0 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T05:17:41.842 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ea419eb1-956c-429a-983a-c66e709dfef8 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711430261343823, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711430261344220, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711430261, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ea419eb1-956c-429a-983a-c66e709dfef8", "db_session_id": "Q1G6QCFCJA951B5S2N99", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T05:17:41.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711430261344300, "job": 1, "event": "recovery_finished"} 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563d05aa8000 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: DB pointer 0x563d05a92000 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** DB Stats ** 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T05:17:41.844 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T05:17:41.845 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative compaction: 0.00 GB write, 0.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval compaction: 0.00 GB write, 0.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache BinnedLRUCache@0x563d059b7090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: using public_addrv [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: starting mon.c rank -1 at public addrs [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] at bind addrs [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:17:41.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(???) e0 preinit fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout: e1 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout: legacy client fscid: -1 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout: No filesystems configured 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T05:17:41.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1265511790' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:41.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1265511790' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi164:172.21.15.164=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: Saving service mon spec with placement smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi164:172.21.15.164=b;count:3 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: Deploying daemon mon.b on smithi164 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/3181731311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.a calling monitor election 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.b calling monitor election 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: fsmap 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mgrmap e13: y(active, since 41s) 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: overall HEALTH_OK 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:41.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:41.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:41.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: Deploying daemon mon.c on smithi136 2024-03-26T05:17:41.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/2364557010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:17:41.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:41.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:41.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 ceph-mon[38106]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T05:17:41.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 bash[38087]: 9a92e0eeb3f7ab6b9dd342808b0509df18b4c93dd538996d81f337c9a3b53c83 2024-03-26T05:17:41.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:41 smithi136 systemd[1]: Started Ceph mon.c for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:17:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:17:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: mon.a calling monitor election 2024-03-26T05:17:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: mon.b calling monitor election 2024-03-26T05:17:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: mon.c calling monitor election 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: fsmap 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: mgrmap e13: y(active, since 49s) 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: overall HEALTH_OK 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: mon.a calling monitor election 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: mon.b calling monitor election 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: mon.c calling monitor election 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: fsmap 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: mgrmap e13: y(active, since 49s) 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: overall HEALTH_OK 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: mon.a calling monitor election 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:17:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: mon.b calling monitor election 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: mon.c calling monitor election 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: fsmap 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: mgrmap e13: y(active, since 49s) 2024-03-26T05:17:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: overall HEALTH_OK 2024-03-26T05:17:46.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:46.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:47.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:47 smithi136 ceph-mon[29560]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:47.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:47.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:47.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:47.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:47 smithi136 ceph-mon[38106]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:47.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:47.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:47.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:47.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.769+0000 7f13b9b09700 1 -- 172.21.15.164:0/2115676122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f139c006240 msgr2=0x7f139c0086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:47.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.769+0000 7f13b9b09700 1 --2- 172.21.15.164:0/2115676122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f139c006240 0x7f139c0086b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f13b4102c20 tx=0x7f13a400fe90 comp rx=0 tx=0).stop 2024-03-26T05:17:47.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.769+0000 7f13b9b09700 1 -- 172.21.15.164:0/2115676122 shutdown_connections 2024-03-26T05:17:47.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.769+0000 7f13b9b09700 1 --2- 172.21.15.164:0/2115676122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f139c006240 0x7f139c0086b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:47.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.769+0000 7f13b9b09700 1 -- 172.21.15.164:0/2115676122 >> 172.21.15.164:0/2115676122 conn(0x7f13b40fd650 msgr2=0x7f13b40ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:47.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.769+0000 7f13b9b09700 1 -- 172.21.15.164:0/2115676122 shutdown_connections 2024-03-26T05:17:47.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.769+0000 7f13b9b09700 1 -- 172.21.15.164:0/2115676122 wait complete. 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.770+0000 7f13b9b09700 1 Processor -- start 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.770+0000 7f13b9b09700 1 -- start start 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.770+0000 7f13b9b09700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f139c006240 0x7f13b4196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.770+0000 7f13b9b09700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13b4197420 0x7f13b419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.770+0000 7f13b9b09700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13b419bca0 0x7f13b419c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.770+0000 7f13b9b09700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4197b10 con 0x7f13b4197420 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.770+0000 7f13b9b09700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b41a3a10 con 0x7f13b419bca0 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.770+0000 7f13b9b09700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b41a3b50 con 0x7f139c006240 2024-03-26T05:17:47.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13b419bca0 0x7f13b419c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13b4197420 0x7f13b419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13b419bca0 0x7f13b419c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:51700/0 (socket says 172.21.15.164:51700) 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13b4197420 0x7f13b419b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.164:34862/0 (socket says 172.21.15.164:34862) 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 -- 172.21.15.164:0/1527498719 learned_addr learned my addr 172.21.15.164:0/1527498719 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b37fe700 1 --2- 172.21.15.164:0/1527498719 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f139c006240 0x7f13b4196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 -- 172.21.15.164:0/1527498719 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f139c006240 msgr2=0x7f13b4196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 --2- 172.21.15.164:0/1527498719 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f139c006240 0x7f13b4196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 -- 172.21.15.164:0/1527498719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13b4197420 msgr2=0x7f13b419b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 --2- 172.21.15.164:0/1527498719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13b4197420 0x7f13b419b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 -- 172.21.15.164:0/1527498719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a400f8e0 con 0x7f13b419bca0 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b3fff700 1 --2- 172.21.15.164:0/1527498719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13b419bca0 0x7f13b419c150 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00ba50 tx=0x7f13ac00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b0ff9700 1 -- 172.21.15.164:0/1527498719 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac00c930 con 0x7f13b419bca0 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b0ff9700 1 -- 172.21.15.164:0/1527498719 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f13ac014440 con 0x7f13b419bca0 2024-03-26T05:17:47.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b0ff9700 1 -- 172.21.15.164:0/1527498719 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0075b0 con 0x7f13b419bca0 2024-03-26T05:17:47.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b41a3e30 con 0x7f13b419bca0 2024-03-26T05:17:47.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.771+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b41a4250 con 0x7f13b419bca0 2024-03-26T05:17:47.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.772+0000 7f13b0ff9700 1 -- 172.21.15.164:0/1527498719 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f13ac007710 con 0x7f13b419bca0 2024-03-26T05:17:47.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.772+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b40636b0 con 0x7f13b419bca0 2024-03-26T05:17:47.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.772+0000 7f13b0ff9700 1 --2- 172.21.15.164:0/1527498719 >> v2:172.21.15.136:6800/2365933975 conn(0x7f13a00393d0 0x7f13a003b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:47.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.772+0000 7f13b0ff9700 1 -- 172.21.15.164:0/1527498719 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f13ac04b250 con 0x7f13b419bca0 2024-03-26T05:17:47.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.773+0000 7f13b37fe700 1 --2- 172.21.15.164:0/1527498719 >> v2:172.21.15.136:6800/2365933975 conn(0x7f13a00393d0 0x7f13a003b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:47.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.773+0000 7f13b37fe700 1 --2- 172.21.15.164:0/1527498719 >> v2:172.21.15.136:6800/2365933975 conn(0x7f13a00393d0 0x7f13a003b890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f13a4010010 tx=0x7f13a4005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:47.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:47.778+0000 7f13b0ff9700 1 -- 172.21.15.164:0/1527498719 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f13ac01b8b0 con 0x7f13b419bca0 2024-03-26T05:17:47.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:47 smithi164 ceph-mon[33554]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:17:48.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.072+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13b41090e0 con 0x7f13b419bca0 2024-03-26T05:17:48.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.072+0000 7f13b0ff9700 1 -- 172.21.15.164:0/1527498719 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f13ac012340 con 0x7f13b419bca0 2024-03-26T05:17:48.062 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T05:17:48.063 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":3,"fsid":"d69b39a8-eb2f-11ee-95d0-87774f69a715","modified":"2024-03-26T05:17:41.364688Z","created":"2024-03-26T05:15:52.811312Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3301","nonce":0},{"type":"v1","addr":"172.21.15.136:6790","nonce":0}]},"addr":"172.21.15.136:6790/0","public_addr":"172.21.15.136:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 >> v2:172.21.15.136:6800/2365933975 conn(0x7f13a00393d0 msgr2=0x7f13a003b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 --2- 172.21.15.164:0/1527498719 >> v2:172.21.15.136:6800/2365933975 conn(0x7f13a00393d0 0x7f13a003b890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f13a4010010 tx=0x7f13a4005da0 comp rx=0 tx=0).stop 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13b419bca0 msgr2=0x7f13b419c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 --2- 172.21.15.164:0/1527498719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13b419bca0 0x7f13b419c150 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00ba50 tx=0x7f13ac00bd60 comp rx=0 tx=0).stop 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 shutdown_connections 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 --2- 172.21.15.164:0/1527498719 >> v2:172.21.15.136:6800/2365933975 conn(0x7f13a00393d0 0x7f13a003b890 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 --2- 172.21.15.164:0/1527498719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13b419bca0 0x7f13b419c150 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 --2- 172.21.15.164:0/1527498719 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f139c006240 0x7f13b4196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 --2- 172.21.15.164:0/1527498719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13b4197420 0x7f13b419b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.074+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 >> 172.21.15.164:0/1527498719 conn(0x7f13b40fd650 msgr2=0x7f13b41057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.075+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 shutdown_connections 2024-03-26T05:17:48.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:48.075+0000 7f13b9b09700 1 -- 172.21.15.164:0/1527498719 wait complete. 2024-03-26T05:17:48.066 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 3 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/1527498719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:48.762 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T05:17:48.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T05:17:48.920 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/1527498719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:48.922 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/1527498719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:48.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:49.880 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:49.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[29560]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T05:17:49.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[29560]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T05:17:49.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[29560]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:49.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[29560]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:49.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[29560]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T05:17:49.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[29560]: Reconfiguring daemon mon.a on smithi136 2024-03-26T05:17:49.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[29560]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:49.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[38106]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T05:17:49.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[38106]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T05:17:49.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[38106]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:49.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[38106]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:49.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[38106]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T05:17:49.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[38106]: Reconfiguring daemon mon.a on smithi136 2024-03-26T05:17:49.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:49 smithi136 ceph-mon[38106]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:49.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:49 smithi164 ceph-mon[33554]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T05:17:49.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:49 smithi164 ceph-mon[33554]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T05:17:49.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:49 smithi164 ceph-mon[33554]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:49.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:49 smithi164 ceph-mon[33554]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:17:49.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:49 smithi164 ceph-mon[33554]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T05:17:49.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:49 smithi164 ceph-mon[33554]: Reconfiguring daemon mon.a on smithi136 2024-03-26T05:17:49.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:49 smithi164 ceph-mon[33554]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[29560]: Reconfiguring mon.c (monmap changed)... 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[29560]: Reconfiguring daemon mon.c on smithi136 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[29560]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:50.945 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[38106]: Reconfiguring mon.c (monmap changed)... 2024-03-26T05:17:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[38106]: Reconfiguring daemon mon.c on smithi136 2024-03-26T05:17:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:50 smithi136 ceph-mon[38106]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:51.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:50 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:50 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:50 smithi164 ceph-mon[33554]: Reconfiguring mon.c (monmap changed)... 2024-03-26T05:17:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:50 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:50 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:50 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:50 smithi164 ceph-mon[33554]: Reconfiguring daemon mon.c on smithi136 2024-03-26T05:17:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:50 smithi164 ceph-mon[33554]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:52.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 -- 172.21.15.136:0/1235012855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f018006b1d0 msgr2=0x7f0180106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:52.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 --2- 172.21.15.136:0/1235012855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f018006b1d0 0x7f0180106cd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0174009990 tx=0x7f0174009ca0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 -- 172.21.15.136:0/1235012855 shutdown_connections 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 --2- 172.21.15.136:0/1235012855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0180107400 0x7f018010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 --2- 172.21.15.136:0/1235012855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f018006a820 0x7f018006ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 --2- 172.21.15.136:0/1235012855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f018006b1d0 0x7f0180106cd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 -- 172.21.15.136:0/1235012855 >> 172.21.15.136:0/1235012855 conn(0x7f0180076fd0 msgr2=0x7f01800773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 -- 172.21.15.136:0/1235012855 shutdown_connections 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.396+0000 7f0186609700 1 -- 172.21.15.136:0/1235012855 wait complete. 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0186609700 1 Processor -- start 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0186609700 1 -- start start 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0186609700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f018006a820 0x7f018019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0186609700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f018006b1d0 0x7f018019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0186609700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0180107400 0x7f0180197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:52.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0186609700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0180198150 con 0x7f0180107400 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0186609700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0180198290 con 0x7f018006b1d0 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0186609700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0180198450 con 0x7f018006a820 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017ffff700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f018006a820 0x7f018019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017ffff700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f018006a820 0x7f018019d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:53208/0 (socket says 172.21.15.136:53208) 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f0184ba6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0180107400 0x7f0180197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017ffff700 1 -- 172.21.15.136:0/1468129177 learned_addr learned my addr 172.21.15.136:0/1468129177 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017f7fe700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f018006b1d0 0x7f018019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017ffff700 1 -- 172.21.15.136:0/1468129177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f018006b1d0 msgr2=0x7f018019de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017ffff700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f018006b1d0 0x7f018019de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017ffff700 1 -- 172.21.15.136:0/1468129177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0180107400 msgr2=0x7f0180197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017ffff700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0180107400 0x7f0180197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.397+0000 7f017ffff700 1 -- 172.21.15.136:0/1468129177 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0170009520 con 0x7f018006a820 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.398+0000 7f0184ba6700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0180107400 0x7f0180197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:17:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.398+0000 7f017ffff700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f018006a820 0x7f018019d930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f017000eb30 tx=0x7f017000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:52.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.398+0000 7f017d7fa700 1 -- 172.21.15.136:0/1468129177 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0170009880 con 0x7f018006a820 2024-03-26T05:17:52.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.398+0000 7f017d7fa700 1 -- 172.21.15.136:0/1468129177 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f01700044d0 con 0x7f018006a820 2024-03-26T05:17:52.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.398+0000 7f017d7fa700 1 -- 172.21.15.136:0/1468129177 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0170003d10 con 0x7f018006a820 2024-03-26T05:17:52.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.398+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01740095f0 con 0x7f018006a820 2024-03-26T05:17:52.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.398+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01801a4800 con 0x7f018006a820 2024-03-26T05:17:52.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.399+0000 7f017d7fa700 1 -- 172.21.15.136:0/1468129177 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f0170004020 con 0x7f018006a820 2024-03-26T05:17:52.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.399+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0180068d40 con 0x7f018006a820 2024-03-26T05:17:52.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.399+0000 7f017d7fa700 1 --2- 172.21.15.136:0/1468129177 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0168039180 0x7f016803b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:52.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.399+0000 7f017d7fa700 1 -- 172.21.15.136:0/1468129177 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f0170014070 con 0x7f018006a820 2024-03-26T05:17:52.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.399+0000 7f017f7fe700 1 --2- 172.21.15.136:0/1468129177 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0168039180 0x7f016803b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:52.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.399+0000 7f017f7fe700 1 --2- 172.21.15.136:0/1468129177 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0168039180 0x7f016803b640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0174006010 tx=0x7f0174005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:52.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.404+0000 7f017d7fa700 1 -- 172.21.15.136:0/1468129177 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f017000fc50 con 0x7f018006a820 2024-03-26T05:17:52.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.600+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f01801117d0 con 0x7f018006a820 2024-03-26T05:17:52.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.600+0000 7f017d7fa700 1 -- 172.21.15.136:0/1468129177 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f017000fc50 con 0x7f018006a820 2024-03-26T05:17:52.591 INFO:teuthology.orchestra.run.smithi136.stdout:# minimal ceph.conf for d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:17:52.591 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-03-26T05:17:52.591 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:17:52.591 INFO:teuthology.orchestra.run.smithi136.stdout: mon_host = [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] 2024-03-26T05:17:52.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.602+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0168039180 msgr2=0x7f016803b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:52.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.602+0000 7f0186609700 1 --2- 172.21.15.136:0/1468129177 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0168039180 0x7f016803b640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0174006010 tx=0x7f0174005960 comp rx=0 tx=0).stop 2024-03-26T05:17:52.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f018006a820 msgr2=0x7f018019d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:52.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f018006a820 0x7f018019d930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f017000eb30 tx=0x7f017000ee40 comp rx=0 tx=0).stop 2024-03-26T05:17:52.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 shutdown_connections 2024-03-26T05:17:52.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 --2- 172.21.15.136:0/1468129177 >> v2:172.21.15.136:6800/2365933975 conn(0x7f0168039180 0x7f016803b640 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0180107400 0x7f0180197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f018006a820 0x7f018019d930 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 --2- 172.21.15.136:0/1468129177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f018006b1d0 0x7f018019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:52.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 >> 172.21.15.136:0/1468129177 conn(0x7f0180076fd0 msgr2=0x7f01800fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:52.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 shutdown_connections 2024-03-26T05:17:52.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:17:52.603+0000 7f0186609700 1 -- 172.21.15.136:0/1468129177 wait complete. 2024-03-26T05:17:53.228 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T05:17:53.228 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:17:53.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T05:17:53.260 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:17:53.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:17:53.328 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:17:53.328 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T05:17:53.353 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:53.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:53.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: Reconfiguring mon.b (monmap changed)... 2024-03-26T05:17:53.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:53.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:53.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:53.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: Reconfiguring daemon mon.b on smithi164 2024-03-26T05:17:53.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:53.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1468129177' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:53.358 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:17:53.358 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:17:53.426 INFO:tasks.cephadm:Adding mgr.y on smithi136 2024-03-26T05:17:53.426 INFO:tasks.cephadm:Adding mgr.x on smithi164 2024-03-26T05:17:53.426 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi136=y;smithi164=x' 2024-03-26T05:17:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: Reconfiguring mon.b (monmap changed)... 2024-03-26T05:17:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: Reconfiguring daemon mon.b on smithi164 2024-03-26T05:17:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1468129177' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: Reconfiguring mon.b (monmap changed)... 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: Reconfiguring daemon mon.b on smithi164 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1468129177' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:53.832 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:17:55.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:55.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:55.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:55.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:54 smithi164 ceph-mon[33554]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[38106]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:54 smithi136 ceph-mon[29560]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 -- 172.21.15.164:0/1296667165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4804103c70 msgr2=0x7f480410a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 --2- 172.21.15.164:0/1296667165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4804103c70 0x7f480410a520 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00b0d0 tx=0x7f47fc00b3e0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 -- 172.21.15.164:0/1296667165 shutdown_connections 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 --2- 172.21.15.164:0/1296667165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4804103c70 0x7f480410a520 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 --2- 172.21.15.164:0/1296667165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48041020d0 0x7f48041024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 --2- 172.21.15.164:0/1296667165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48041032d0 0x7f4804103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 -- 172.21.15.164:0/1296667165 >> 172.21.15.164:0/1296667165 conn(0x7f48040fd690 msgr2=0x7f48040ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 -- 172.21.15.164:0/1296667165 shutdown_connections 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 -- 172.21.15.164:0/1296667165 wait complete. 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 Processor -- start 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.274+0000 7f4809f40700 1 -- start start 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4809f40700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48041020d0 0x7f4804199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4809f40700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48041032d0 0x7f4804199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4809f40700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4804103c70 0x7f48041a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4809f40700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48041a1610 con 0x7f48041032d0 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4809f40700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48041a1750 con 0x7f48041020d0 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4809f40700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48041a18e0 con 0x7f4804103c70 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48041020d0 0x7f4804199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48041020d0 0x7f4804199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:51726/0 (socket says 172.21.15.164:51726) 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 -- 172.21.15.164:0/2455681863 learned_addr learned my addr 172.21.15.164:0/2455681863 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4803fff700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4804103c70 0x7f48041a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f47faffd700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48041032d0 0x7f4804199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 -- 172.21.15.164:0/2455681863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4804103c70 msgr2=0x7f48041a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4804103c70 0x7f48041a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 -- 172.21.15.164:0/2455681863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48041032d0 msgr2=0x7f4804199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48041032d0 0x7f4804199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 -- 172.21.15.164:0/2455681863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc009d00 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48037fe700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48041020d0 0x7f4804199a80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f47f400d980 tx=0x7f47f400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48017fa700 1 -- 172.21.15.164:0/2455681863 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f400f890 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48041a1bc0 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48017fa700 1 -- 172.21.15.164:0/2455681863 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f47f4010460 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f48017fa700 1 -- 172.21.15.164:0/2455681863 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f400fea0 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.275+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48041a2110 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.276+0000 7f48017fa700 1 -- 172.21.15.164:0/2455681863 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7f47f40195c0 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.276+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4804068d40 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.276+0000 7f48017fa700 1 --2- 172.21.15.164:0/2455681863 >> v2:172.21.15.136:6800/2365933975 conn(0x7f47f0039180 0x7f47f003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.276+0000 7f48017fa700 1 -- 172.21.15.164:0/2455681863 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f47f404b500 con 0x7f48041020d0 2024-03-26T05:17:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.277+0000 7f47faffd700 1 --2- 172.21.15.164:0/2455681863 >> v2:172.21.15.136:6800/2365933975 conn(0x7f47f0039180 0x7f47f003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:17:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.277+0000 7f47faffd700 1 --2- 172.21.15.164:0/2455681863 >> v2:172.21.15.136:6800/2365933975 conn(0x7f47f0039180 0x7f47f003b640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009be0 tx=0x7f47ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:17:55.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.282+0000 7f48017fa700 1 -- 172.21.15.164:0/2455681863 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f47f4019e30 con 0x7f48041020d0 2024-03-26T05:17:55.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.484+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi136=y;smithi164=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f48040759d0 con 0x7f47f0039180 2024-03-26T05:17:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.490+0000 7f48017fa700 1 -- 172.21.15.164:0/2455681863 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f48040759d0 con 0x7f47f0039180 2024-03-26T05:17:55.480 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled mgr update... 2024-03-26T05:17:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 >> v2:172.21.15.136:6800/2365933975 conn(0x7f47f0039180 msgr2=0x7f47f003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 --2- 172.21.15.164:0/2455681863 >> v2:172.21.15.136:6800/2365933975 conn(0x7f47f0039180 0x7f47f003b640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009be0 tx=0x7f47ec009260 comp rx=0 tx=0).stop 2024-03-26T05:17:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48041020d0 msgr2=0x7f4804199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:17:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48041020d0 0x7f4804199a80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f47f400d980 tx=0x7f47f400dc90 comp rx=0 tx=0).stop 2024-03-26T05:17:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 shutdown_connections 2024-03-26T05:17:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 --2- 172.21.15.164:0/2455681863 >> v2:172.21.15.136:6800/2365933975 conn(0x7f47f0039180 0x7f47f003b640 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48041020d0 0x7f4804199a80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4804103c70 0x7f48041a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 --2- 172.21.15.164:0/2455681863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48041032d0 0x7f4804199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:17:55.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 >> 172.21.15.164:0/2455681863 conn(0x7f48040fd690 msgr2=0x7f48040742b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:17:55.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 shutdown_connections 2024-03-26T05:17:55.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:17:55.492+0000 7f4809f40700 1 -- 172.21.15.164:0/2455681863 wait complete. 2024-03-26T05:17:56.028 DEBUG:teuthology.orchestra.run.smithi164:mgr.x> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T05:17:56.030 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T05:17:56.031 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:17:56.031 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T05:17:56.050 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T05:17:56.050 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-03-26T05:17:56.107 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T05:17:56.107 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T05:17:56.107 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T05:17:56.107 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T05:17:56.108 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T05:17:56.108 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 05:17:40.813111351 +0000 2024-03-26T05:17:56.108 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 05:12:11.507237765 +0000 2024-03-26T05:17:56.108 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 05:12:11.507237765 +0000 2024-03-26T05:17:56.108 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 05:12:11.507237765 +0000 2024-03-26T05:17:56.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T05:17:56.176 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T05:17:56.177 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T05:17:56.177 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000216557 s, 2.4 MB/s 2024-03-26T05:17:56.178 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T05:17:56.236 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-03-26T05:17:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T05:17:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T05:17:56.293 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T05:17:56.293 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T05:17:56.293 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T05:17:56.293 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 05:17:40.813111351 +0000 2024-03-26T05:17:56.293 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 05:12:11.839230076 +0000 2024-03-26T05:17:56.293 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 05:12:11.839230076 +0000 2024-03-26T05:17:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 05:12:11.839230076 +0000 2024-03-26T05:17:56.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T05:17:56.360 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T05:17:56.361 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T05:17:56.361 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000236626 s, 2.2 MB/s 2024-03-26T05:17:56.362 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T05:17:56.419 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-03-26T05:17:56.477 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T05:17:56.477 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T05:17:56.477 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T05:17:56.478 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T05:17:56.478 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T05:17:56.478 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 05:17:40.813111351 +0000 2024-03-26T05:17:56.478 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 05:12:12.190221947 +0000 2024-03-26T05:17:56.478 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 05:12:12.190221947 +0000 2024-03-26T05:17:56.478 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 05:12:12.190221947 +0000 2024-03-26T05:17:56.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T05:17:56.542 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T05:17:56.542 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T05:17:56.542 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000226207 s, 2.3 MB/s 2024-03-26T05:17:56.543 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T05:17:56.599 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-03-26T05:17:56.654 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T05:17:56.654 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T05:17:56.654 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T05:17:56.655 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T05:17:56.655 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T05:17:56.655 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-26 05:17:40.814111329 +0000 2024-03-26T05:17:56.655 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-26 05:12:12.525214188 +0000 2024-03-26T05:17:56.655 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-26 05:12:12.525214188 +0000 2024-03-26T05:17:56.655 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-26 05:12:12.525214188 +0000 2024-03-26T05:17:56.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T05:17:56.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi136=y;smithi164=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: Saving service mgr spec with placement smithi136=y;smithi164=x;count:2 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T05:17:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:56 smithi164 ceph-mon[33554]: Deploying daemon mgr.x on smithi164 2024-03-26T05:17:56.721 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-26T05:17:56.721 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-26T05:17:56.721 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00021927 s, 2.3 MB/s 2024-03-26T05:17:56.722 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T05:17:56.778 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:17:56.779 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T05:17:56.795 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T05:17:56.795 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-03-26T05:17:56.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi136=y;smithi164=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:56.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: Saving service mgr spec with placement smithi136=y;smithi164=x;count:2 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[38106]: Deploying daemon mgr.x on smithi164 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi136=y;smithi164=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:17:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: Saving service mgr spec with placement smithi136=y;smithi164=x;count:2 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:17:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:56 smithi136 ceph-mon[29560]: Deploying daemon mgr.x on smithi164 2024-03-26T05:17:56.850 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T05:17:56.850 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T05:17:56.850 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-26T05:17:56.851 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T05:17:56.851 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T05:17:56.851 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-26 05:17:32.214472847 +0000 2024-03-26T05:17:56.851 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-26 05:12:12.303573904 +0000 2024-03-26T05:17:56.851 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-26 05:12:12.303573904 +0000 2024-03-26T05:17:56.851 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-26 05:12:12.303573904 +0000 2024-03-26T05:17:56.851 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T05:17:56.918 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-26T05:17:56.918 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-26T05:17:56.918 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000273036 s, 1.9 MB/s 2024-03-26T05:17:56.919 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T05:17:56.977 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-03-26T05:17:57.032 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T05:17:57.033 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T05:17:57.033 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T05:17:57.033 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T05:17:57.033 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T05:17:57.033 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-26 05:17:32.215472830 +0000 2024-03-26T05:17:57.033 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-26 05:12:12.644567947 +0000 2024-03-26T05:17:57.033 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-26 05:12:12.644567947 +0000 2024-03-26T05:17:57.033 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-26 05:12:12.644567947 +0000 2024-03-26T05:17:57.033 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T05:17:57.099 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-26T05:17:57.100 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-26T05:17:57.100 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000199102 s, 2.6 MB/s 2024-03-26T05:17:57.101 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T05:17:57.158 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-03-26T05:17:57.215 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T05:17:57.215 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T05:17:57.215 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T05:17:57.215 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T05:17:57.215 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T05:17:57.215 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-26 05:17:32.215472830 +0000 2024-03-26T05:17:57.215 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-26 05:12:12.981562060 +0000 2024-03-26T05:17:57.216 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-26 05:12:12.981562060 +0000 2024-03-26T05:17:57.216 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-26 05:12:12.981562060 +0000 2024-03-26T05:17:57.216 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T05:17:57.363 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-26T05:17:57.363 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-26T05:17:57.363 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000212006 s, 2.4 MB/s 2024-03-26T05:17:57.365 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T05:17:57.380 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-03-26T05:17:57.441 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T05:17:57.441 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T05:17:57.441 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T05:17:57.441 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T05:17:57.441 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T05:17:57.441 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-26 05:17:57.329052226 +0000 2024-03-26T05:17:57.442 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-26 05:12:13.321556121 +0000 2024-03-26T05:17:57.442 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-26 05:12:13.321556121 +0000 2024-03-26T05:17:57.442 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-26 05:12:13.321556121 +0000 2024-03-26T05:17:57.442 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T05:17:57.541 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-26T05:17:57.541 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-26T05:17:57.541 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000206401 s, 2.5 MB/s 2024-03-26T05:17:57.543 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T05:17:57.557 INFO:tasks.cephadm:Deploying osd.0 on smithi136 with /dev/vg_nvme/lv_4... 2024-03-26T05:17:57.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T05:17:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:57 smithi164 ceph-mon[33554]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:57.718 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:57 smithi164 systemd[1]: Starting Ceph mgr.x for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:17:57.794 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:17:57.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:57 smithi136 ceph-mon[29560]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:57.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:57 smithi136 ceph-mon[38106]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:58.035 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:57 smithi164 podman[35423]: 2024-03-26 05:17:57.77590342 +0000 UTC m=+0.104396063 container create 966a6b88a8afe00c01d24bda80db9316f9cd69f692e80f50cf30c714419b52bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T05:17:58.035 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:57 smithi164 podman[35423]: 2024-03-26 05:17:57.689078729 +0000 UTC m=+0.017571396 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:17:58.035 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:57 smithi164 podman[35423]: 2024-03-26 05:17:57.969762207 +0000 UTC m=+0.298254861 container init 966a6b88a8afe00c01d24bda80db9316f9cd69f692e80f50cf30c714419b52bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=squid-5e70124) 2024-03-26T05:17:58.035 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:57 smithi164 podman[35423]: 2024-03-26 05:17:57.972603974 +0000 UTC m=+0.301096627 container start 966a6b88a8afe00c01d24bda80db9316f9cd69f692e80f50cf30c714419b52bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T05:17:58.035 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:57 smithi164 ceph-mgr[35442]: -- 172.21.15.164:0/2317935466 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c7089841a0 con 0x55c70895b000 2024-03-26T05:17:58.035 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:58 smithi164 bash[35423]: 966a6b88a8afe00c01d24bda80db9316f9cd69f692e80f50cf30c714419b52bb 2024-03-26T05:17:58.467 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:58 smithi164 systemd[1]: Started Ceph mgr.x for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:17:58.967 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:17:58.544+0000 7f3cd9d63200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T05:17:58.968 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:17:58.643+0000 7f3cd9d63200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T05:17:59.259 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.259 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.259 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.259 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.259 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:59.259 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-mon[33554]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:59.525 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:17:59.330+0000 7f3cd9d63200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T05:17:59.525 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:17:59.436+0000 7f3cd9d63200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T05:17:59.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[38106]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:17:59.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:17:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:17:59 smithi136 ceph-mon[29560]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:17:59.683 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T05:17:59.683 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T05:17:59.683 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-26T05:17:59.683 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-26T05:17:59.683 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00932176 s, 1.1 GB/s 2024-03-26T05:17:59.683 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-26T05:17:59.967 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:17:59.586+0000 7f3cd9d63200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T05:17:59.968 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:17:59.687+0000 7f3cd9d63200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T05:17:59.968 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:17:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:17:59.777+0000 7f3cd9d63200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T05:18:00.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2024-03-26T05:18:00.338 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:00.050+0000 7f3cd9d63200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T05:18:00.339 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:00.163+0000 7f3cd9d63200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T05:18:00.339 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:00.251+0000 7f3cd9d63200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T05:18:00.339 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:00.348+0000 7f3cd9d63200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T05:18:00.523 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[29560]: Reconfiguring daemon mgr.y on smithi136 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.524 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:00.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:00.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T05:18:00.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T05:18:00.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T05:18:00.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:00.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:00 smithi136 ceph-mon[38106]: Reconfiguring daemon mgr.y on smithi136 2024-03-26T05:18:00.717 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:00.531+0000 7f3cd9d63200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:00 smithi164 ceph-mon[33554]: Reconfiguring daemon mgr.y on smithi136 2024-03-26T05:18:01.306 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:01.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:01 smithi136 ceph-mon[29560]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:01.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:01.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:01.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:01.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:01 smithi136 ceph-mon[38106]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:01.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:01.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:01.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:01.960 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:01.506+0000 7f3cd9d63200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T05:18:01.960 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:01.635+0000 7f3cd9d63200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T05:18:01.960 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:01.968+0000 7f3cd9d63200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T05:18:01.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:01 smithi164 ceph-mon[33554]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:01.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:01.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:01.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:02.217 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:02.058+0000 7f3cd9d63200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T05:18:02.560 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:02.254+0000 7f3cd9d63200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:18:02.560 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:02.437+0000 7f3cd9d63200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T05:18:02.683 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:02 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:02.683 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:02 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:02.684 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:02 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:02.684 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:02 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:02.684 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:02 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:02.684 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:02 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:02.684 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:02 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:02.684 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:02 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:02.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 -- 172.21.15.136:0/3340229696 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 msgr2=0x7fb654074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:02.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 --2- 172.21.15.136:0/3340229696 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 0x7fb654074490 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb6440098f0 tx=0x7fb644009c00 comp rx=0 tx=0).stop 2024-03-26T05:18:02.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 -- 172.21.15.136:0/3340229696 shutdown_connections 2024-03-26T05:18:02.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 --2- 172.21.15.136:0/3340229696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654109fd0 0x7fb65410c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:02.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 --2- 172.21.15.136:0/3340229696 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 0x7fb654074490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:02.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 --2- 172.21.15.136:0/3340229696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb654074a60 0x7fb654074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:02.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 -- 172.21.15.136:0/3340229696 >> 172.21.15.136:0/3340229696 conn(0x7fb6540fd690 msgr2=0x7fb6540ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:18:02.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 -- 172.21.15.136:0/3340229696 shutdown_connections 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 -- 172.21.15.136:0/3340229696 wait complete. 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.915+0000 7fb659588700 1 Processor -- start 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb659588700 1 -- start start 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb659588700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654074a60 0x7fb654199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb659588700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 0x7fb654199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb659588700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb654109fd0 0x7fb6541a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb659588700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6541a1500 con 0x7fb654074a60 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb659588700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6541a1640 con 0x7fb654109fd0 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb659588700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6541a17d0 con 0x7fb654076030 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb64a7fc700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 0x7fb654199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654074a60 0x7fb654199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654074a60 0x7fb654199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33964/0 (socket says 172.21.15.136:33964) 2024-03-26T05:18:02.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 -- 172.21.15.136:0/301264570 learned_addr learned my addr 172.21.15.136:0/301264570 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb64a7fc700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 0x7fb654199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:35642/0 (socket says 172.21.15.136:35642) 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb6537fe700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb654109fd0 0x7fb6541a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 -- 172.21.15.136:0/301264570 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 msgr2=0x7fb654199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 0x7fb654199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 -- 172.21.15.136:0/301264570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb654109fd0 msgr2=0x7fb6541a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb654109fd0 0x7fb6541a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 -- 172.21.15.136:0/301264570 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c009d00 con 0x7fb654074a60 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb64a7fc700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 0x7fb654199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.916+0000 7fb652ffd700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654074a60 0x7fb654199970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb644009230 tx=0x7fb64400b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.917+0000 7fb650ff9700 1 -- 172.21.15.136:0/301264570 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64401c070 con 0x7fb654074a60 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.917+0000 7fb650ff9700 1 -- 172.21.15.136:0/301264570 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fb6440054f0 con 0x7fb654074a60 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.917+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb644009520 con 0x7fb654074a60 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.917+0000 7fb650ff9700 1 -- 172.21.15.136:0/301264570 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644004c90 con 0x7fb654074a60 2024-03-26T05:18:02.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.917+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6541a1cd0 con 0x7fb654074a60 2024-03-26T05:18:02.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.917+0000 7fb650ff9700 1 -- 172.21.15.136:0/301264570 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45950+0+0 (secure 0 0 0) 0x7fb644029400 con 0x7fb654074a60 2024-03-26T05:18:02.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.918+0000 7fb650ff9700 1 --2- 172.21.15.136:0/301264570 >> v2:172.21.15.136:6800/2365933975 conn(0x7fb640039180 0x7fb64003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:02.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.918+0000 7fb650ff9700 1 -- 172.21.15.136:0/301264570 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fb64404cf90 con 0x7fb654074a60 2024-03-26T05:18:02.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.918+0000 7fb64a7fc700 1 --2- 172.21.15.136:0/301264570 >> v2:172.21.15.136:6800/2365933975 conn(0x7fb640039180 0x7fb64003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:02.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.918+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6541086d0 con 0x7fb654074a60 2024-03-26T05:18:02.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.918+0000 7fb64a7fc700 1 --2- 172.21.15.136:0/301264570 >> v2:172.21.15.136:6800/2365933975 conn(0x7fb640039180 0x7fb64003b640 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb63c005950 tx=0x7fb63c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:18:02.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:02.923+0000 7fb650ff9700 1 -- 172.21.15.136:0/301264570 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb644004df0 con 0x7fb654074a60 2024-03-26T05:18:02.967 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:02.719+0000 7f3cd9d63200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:18:02.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:02.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:02.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:02.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:03.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:03.126+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb65410f430 con 0x7fb640039180 2024-03-26T05:18:03.467 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:02.979+0000 7f3cd9d63200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T05:18:03.468 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:03.213+0000 7f3cd9d63200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T05:18:03.819 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:03 smithi136 ceph-mon[29560]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:03.819 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:03 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:03.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:03 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:03.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:03 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:03.820 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:03 smithi136 ceph-mon[38106]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:03.820 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:03 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:03.820 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:03 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:03.820 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:03 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:03.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:03 smithi164 ceph-mon[33554]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:03 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:03 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:03 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:04.467 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:04.017+0000 7f3cd9d63200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T05:18:04.695 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:04 smithi136 ceph-mon[29560]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:04.695 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:04 smithi136 ceph-mon[38106]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:04.885 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:04 smithi164 ceph-mon[33554]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:04.886 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:04.586+0000 7f3cd9d63200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T05:18:04.886 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:04.683+0000 7f3cd9d63200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T05:18:04.886 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:18:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:18:04.894+0000 7f3cd9d63200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T05:18:05.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:05.588+0000 7fb650ff9700 1 -- 172.21.15.136:0/301264570 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mgrmap(e 14) v1 ==== 91308+0+0 (secure 0 0 0) 0x7fb64401fc90 con 0x7fb654074a60 2024-03-26T05:18:05.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[29560]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:05.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[29560]: Standby manager daemon x started 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[29560]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[29560]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[29560]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[29560]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[38106]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[38106]: Standby manager daemon x started 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[38106]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[38106]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:18:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[38106]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:18:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:05 smithi136 ceph-mon[38106]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:18:05.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:05 smithi164 ceph-mon[33554]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:05.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:05 smithi164 ceph-mon[33554]: Standby manager daemon x started 2024-03-26T05:18:05.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:05 smithi164 ceph-mon[33554]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:18:05.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:05 smithi164 ceph-mon[33554]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:18:05.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:05 smithi164 ceph-mon[33554]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:18:05.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:05 smithi164 ceph-mon[33554]: from='mgr.? 172.21.15.164:0/2272405500' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:18:06.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:06 smithi164 ceph-mon[33554]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T05:18:06.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:18:06.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3301248996' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4698473-d228-4853-b404-54b255f0a9d3"}]: dispatch 2024-03-26T05:18:06.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3301248996' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4698473-d228-4853-b404-54b255f0a9d3"}]': finished 2024-03-26T05:18:06.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:06 smithi164 ceph-mon[33554]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T05:18:06.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:06.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1109370503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:07.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[29560]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3301248996' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4698473-d228-4853-b404-54b255f0a9d3"}]: dispatch 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3301248996' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4698473-d228-4853-b404-54b255f0a9d3"}]': finished 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[29560]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1109370503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[38106]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:18:07.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3301248996' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4698473-d228-4853-b404-54b255f0a9d3"}]: dispatch 2024-03-26T05:18:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3301248996' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4698473-d228-4853-b404-54b255f0a9d3"}]': finished 2024-03-26T05:18:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[38106]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T05:18:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1109370503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:07.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:07 smithi164 ceph-mon[33554]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:08.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:07 smithi136 ceph-mon[29560]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:08.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:07 smithi136 ceph-mon[38106]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:09.934 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:09 smithi136 ceph-mon[29560]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:09.934 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:09 smithi136 ceph-mon[38106]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:09.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:09 smithi164 ceph-mon[33554]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:11.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:11 smithi136 ceph-mon[38106]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:11.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:11 smithi136 ceph-mon[29560]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:11.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:11 smithi164 ceph-mon[33554]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:12.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:12 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T05:18:12.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:12 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:12.917 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T05:18:12.917 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:12 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:12.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:12 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T05:18:12.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:12 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:13.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:13 smithi136 ceph-mon[29560]: Deploying daemon osd.0 on smithi136 2024-03-26T05:18:13.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:13 smithi136 ceph-mon[29560]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:13.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:13 smithi136 ceph-mon[38106]: Deploying daemon osd.0 on smithi136 2024-03-26T05:18:13.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:13 smithi136 ceph-mon[38106]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:13.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:13 smithi164 ceph-mon[33554]: Deploying daemon osd.0 on smithi136 2024-03-26T05:18:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:13 smithi164 ceph-mon[33554]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:15.879 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:15 smithi136 ceph-mon[29560]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:15.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:15 smithi136 ceph-mon[38106]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:15.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:15 smithi164 ceph-mon[33554]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:17.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:17 smithi136 ceph-mon[38106]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:17.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:17.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:17.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:17.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:17 smithi136 ceph-mon[29560]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:17.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:17.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:17.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:17.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:17 smithi164 ceph-mon[33554]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:17.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:17.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:17.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:19.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:19 smithi136 ceph-mon[29560]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:19 smithi136 ceph-mon[38106]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:19.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:19 smithi164 ceph-mon[33554]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:21.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.174+0000 7fb650ff9700 1 -- 172.21.15.136:0/301264570 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb65410f430 con 0x7fb640039180 2024-03-26T05:18:21.164 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 0 on host 'smithi136' 2024-03-26T05:18:21.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 >> v2:172.21.15.136:6800/2365933975 conn(0x7fb640039180 msgr2=0x7fb64003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:21.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 --2- 172.21.15.136:0/301264570 >> v2:172.21.15.136:6800/2365933975 conn(0x7fb640039180 0x7fb64003b640 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb63c005950 tx=0x7fb63c00b410 comp rx=0 tx=0).stop 2024-03-26T05:18:21.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654074a60 msgr2=0x7fb654199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:21.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654074a60 0x7fb654199970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb644009230 tx=0x7fb64400b560 comp rx=0 tx=0).stop 2024-03-26T05:18:21.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 shutdown_connections 2024-03-26T05:18:21.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 --2- 172.21.15.136:0/301264570 >> v2:172.21.15.136:6800/2365933975 conn(0x7fb640039180 0x7fb64003b640 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:21.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb654109fd0 0x7fb6541a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:21.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb654074a60 0x7fb654199970 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:21.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 --2- 172.21.15.136:0/301264570 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb654076030 0x7fb654199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:21.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 >> 172.21.15.136:0/301264570 conn(0x7fb6540fd690 msgr2=0x7fb6540ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:18:21.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 shutdown_connections 2024-03-26T05:18:21.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:21.176+0000 7fb659588700 1 -- 172.21.15.136:0/301264570 wait complete. 2024-03-26T05:18:21.465 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.465 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.465 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:21.465 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:21.465 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:21.466 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T05:18:21.467 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.467 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.467 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:21 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:21.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:21 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:21.781 DEBUG:teuthology.orchestra.run.smithi136:osd.0> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T05:18:21.783 INFO:tasks.cephadm:Deploying osd.1 on smithi136 with /dev/vg_nvme/lv_3... 2024-03-26T05:18:21.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T05:18:22.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-mon[29560]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T05:18:22.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-mon[29560]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T05:18:22.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-mon[29560]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:18:22.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:22.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-mon[38106]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T05:18:22.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-mon[38106]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T05:18:22.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-mon[38106]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:18:22.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:22.324 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0[42137]: 2024-03-26T05:18:22.259+0000 7f01ea3d1700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T05:18:22.324 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:18:22 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0[42137]: 2024-03-26T05:18:22.263+0000 7f01e61c4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:18:22.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:22 smithi164 ceph-mon[33554]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T05:18:22.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:22 smithi164 ceph-mon[33554]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T05:18:22.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:22 smithi164 ceph-mon[33554]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:18:22.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:22 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:23.002 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:23.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[38106]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[38106]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[38106]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[29560]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[29560]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:23.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:23 smithi136 ceph-mon[29560]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:23.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:23 smithi164 ceph-mon[33554]: from='osd.0 v2:172.21.15.136:6801/1923051194' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:18:23.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:23 smithi164 ceph-mon[33554]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T05:18:23.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:23 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:23.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:23 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:23.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:23 smithi164 ceph-mon[33554]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[38106]: purged_snaps scrub starts 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[38106]: purged_snaps scrub ok 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[38106]: osd.0 v2:172.21.15.136:6801/1923051194 boot 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[38106]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[29560]: purged_snaps scrub starts 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[29560]: purged_snaps scrub ok 2024-03-26T05:18:24.386 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:24.387 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[29560]: osd.0 v2:172.21.15.136:6801/1923051194 boot 2024-03-26T05:18:24.387 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[29560]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T05:18:24.387 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:24 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:24 smithi164 ceph-mon[33554]: purged_snaps scrub starts 2024-03-26T05:18:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:24 smithi164 ceph-mon[33554]: purged_snaps scrub ok 2024-03-26T05:18:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:24 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:24 smithi164 ceph-mon[33554]: osd.0 v2:172.21.15.136:6801/1923051194 boot 2024-03-26T05:18:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:24 smithi164 ceph-mon[33554]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T05:18:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:24 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:18:25.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:25 smithi136 ceph-mon[29560]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T05:18:25.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:25 smithi136 ceph-mon[29560]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:25.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:25 smithi136 ceph-mon[38106]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T05:18:25.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:25 smithi136 ceph-mon[38106]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:25.691 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T05:18:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T05:18:25.692 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-26T05:18:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-26T05:18:25.692 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00863639 s, 1.2 GB/s 2024-03-26T05:18:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-26T05:18:25.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:25 smithi164 ceph-mon[33554]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T05:18:25.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:25 smithi164 ceph-mon[33554]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:26.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2024-03-26T05:18:26.721 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:27.157 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: Detected new or changed devices on smithi136 2024-03-26T05:18:27.157 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi136 to 16064M 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[38106]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: Detected new or changed devices on smithi136 2024-03-26T05:18:27.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.159 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.159 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:27.159 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi136 to 16064M 2024-03-26T05:18:27.159 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.159 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:27.159 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:27.159 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.159 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:26 smithi136 ceph-mon[29560]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: Detected new or changed devices on smithi136 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi136 to 16064M 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:26 smithi164 ceph-mon[33554]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 -- 172.21.15.136:0/2626727378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a40103c50 msgr2=0x7f3a4010a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 --2- 172.21.15.136:0/2626727378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a40103c50 0x7f3a4010a500 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009f90 tx=0x7f3a3800b330 comp rx=0 tx=0).stop 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 -- 172.21.15.136:0/2626727378 shutdown_connections 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 --2- 172.21.15.136:0/2626727378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a401020b0 0x7f3a401024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 --2- 172.21.15.136:0/2626727378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a40103c50 0x7f3a4010a500 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 --2- 172.21.15.136:0/2626727378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a401032b0 0x7f3a40103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 -- 172.21.15.136:0/2626727378 >> 172.21.15.136:0/2626727378 conn(0x7f3a400fd690 msgr2=0x7f3a400ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 -- 172.21.15.136:0/2626727378 shutdown_connections 2024-03-26T05:18:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.292+0000 7f3a484e9700 1 -- 172.21.15.136:0/2626727378 wait complete. 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.293+0000 7f3a484e9700 1 Processor -- start 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.293+0000 7f3a484e9700 1 -- start start 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.293+0000 7f3a484e9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a401020b0 0x7f3a401978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.293+0000 7f3a484e9700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a401032b0 0x7f3a40197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.293+0000 7f3a484e9700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a40103c50 0x7f3a4019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.293+0000 7f3a484e9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4019f460 con 0x7f3a401020b0 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.293+0000 7f3a484e9700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4019f5a0 con 0x7f3a40103c50 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.293+0000 7f3a484e9700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4019f730 con 0x7f3a401032b0 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a46285700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a401020b0 0x7f3a401978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a46285700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a401020b0 0x7f3a401978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60106/0 (socket says 172.21.15.136:60106) 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a45a84700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a401032b0 0x7f3a40197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a46285700 1 -- 172.21.15.136:0/512331417 learned_addr learned my addr 172.21.15.136:0/512331417 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a46a86700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a40103c50 0x7f3a4019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a45a84700 1 -- 172.21.15.136:0/512331417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a40103c50 msgr2=0x7f3a4019ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a45a84700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a40103c50 0x7f3a4019ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a45a84700 1 -- 172.21.15.136:0/512331417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a401020b0 msgr2=0x7f3a401978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a45a84700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a401020b0 0x7f3a401978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a45a84700 1 -- 172.21.15.136:0/512331417 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a300095f0 con 0x7f3a401032b0 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a46a86700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a40103c50 0x7f3a4019ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a45a84700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a401032b0 0x7f3a40197e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000ebc0 tx=0x7f3a3000eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a377fe700 1 -- 172.21.15.136:0/512331417 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000ccf0 con 0x7f3a401032b0 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a377fe700 1 -- 172.21.15.136:0/512331417 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3a3000ce50 con 0x7f3a401032b0 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a377fe700 1 -- 172.21.15.136:0/512331417 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30010430 con 0x7f3a401032b0 2024-03-26T05:18:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a38009c40 con 0x7f3a401032b0 2024-03-26T05:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.294+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4019fdd0 con 0x7f3a401032b0 2024-03-26T05:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.295+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40108c90 con 0x7f3a401032b0 2024-03-26T05:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.296+0000 7f3a377fe700 1 -- 172.21.15.136:0/512331417 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 14) v1 ==== 91308+0+0 (secure 0 0 0) 0x7f3a30004020 con 0x7f3a401032b0 2024-03-26T05:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.296+0000 7f3a377fe700 1 --2- 172.21.15.136:0/512331417 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3a2c06d900 0x7f3a2c06fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.296+0000 7f3a46285700 1 --2- 172.21.15.136:0/512331417 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3a2c06d900 0x7f3a2c06fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.296+0000 7f3a377fe700 1 -- 172.21.15.136:0/512331417 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1541+0+0 (secure 0 0 0) 0x7f3a30014070 con 0x7f3a401032b0 2024-03-26T05:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.296+0000 7f3a46285700 1 --2- 172.21.15.136:0/512331417 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3a2c06d900 0x7f3a2c06fdc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009aa0 tx=0x7f3a3c009240 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.301+0000 7f3a377fe700 1 -- 172.21.15.136:0/512331417 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3a30057040 con 0x7f3a401032b0 2024-03-26T05:18:28.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:28.507+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a400624c0 con 0x7f3a2c06d900 2024-03-26T05:18:28.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:28 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:28.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:28 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:28.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:28 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:28.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:28 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:28.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:28 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:28.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:28 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:28.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:28 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:28.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:28 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:28.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:28 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:29.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:29 smithi136 ceph-mon[29560]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:29.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:29 smithi136 ceph-mon[29560]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:29 smithi136 ceph-mon[38106]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:29 smithi136 ceph-mon[38106]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:29.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:29 smithi164 ceph-mon[33554]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:29.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:29 smithi164 ceph-mon[33554]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:31.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[38106]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:31.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2004098359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]: dispatch 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]: dispatch 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]': finished 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[38106]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[29560]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2004098359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]: dispatch 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]: dispatch 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]': finished 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[29560]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T05:18:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:31.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:31 smithi164 ceph-mon[33554]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:31.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2004098359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]: dispatch 2024-03-26T05:18:31.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]: dispatch 2024-03-26T05:18:31.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800f61d3-4725-45b6-af8c-0653cb8b4c82"}]': finished 2024-03-26T05:18:31.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:31 smithi164 ceph-mon[33554]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T05:18:31.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:32.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3484796075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:32.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3484796075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:32.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3484796075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:33.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:33 smithi136 ceph-mon[29560]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:33.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:33 smithi136 ceph-mon[38106]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:33.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:33 smithi164 ceph-mon[33554]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:35.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:35 smithi136 ceph-mon[29560]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:35.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:35 smithi136 ceph-mon[38106]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:35 smithi164 ceph-mon[33554]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:37.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:37 smithi136 ceph-mon[29560]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:37.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:37 smithi136 ceph-mon[38106]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:37.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:37 smithi164 ceph-mon[33554]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:38.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:38 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T05:18:38.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:38 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:38.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:38 smithi136 ceph-mon[29560]: Deploying daemon osd.1 on smithi136 2024-03-26T05:18:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:38 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T05:18:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:38 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:38 smithi136 ceph-mon[38106]: Deploying daemon osd.1 on smithi136 2024-03-26T05:18:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:38 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T05:18:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:38 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:38 smithi164 ceph-mon[33554]: Deploying daemon osd.1 on smithi136 2024-03-26T05:18:39.680 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:39 smithi136 ceph-mon[38106]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:39.681 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:39 smithi136 ceph-mon[29560]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:39.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:39 smithi164 ceph-mon[33554]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:41.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:41 smithi164 ceph-mon[33554]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:42.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:41 smithi136 ceph-mon[29560]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:42.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:41 smithi136 ceph-mon[38106]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:43.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:43 smithi136 ceph-mon[29560]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:43.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:43.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:43.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:43.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:43 smithi136 ceph-mon[38106]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:43.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:43.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:43.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:43.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:43 smithi164 ceph-mon[33554]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:43.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:45.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:45 smithi136 ceph-mon[38106]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:45.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:45 smithi136 ceph-mon[29560]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:45.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:45 smithi164 ceph-mon[33554]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:46.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.433+0000 7f3a377fe700 1 -- 172.21.15.136:0/512331417 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3a400624c0 con 0x7f3a2c06d900 2024-03-26T05:18:46.424 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 1 on host 'smithi136' 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.435+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3a2c06d900 msgr2=0x7f3a2c06fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.435+0000 7f3a484e9700 1 --2- 172.21.15.136:0/512331417 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3a2c06d900 0x7f3a2c06fdc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009aa0 tx=0x7f3a3c009240 comp rx=0 tx=0).stop 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.435+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a401032b0 msgr2=0x7f3a40197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.435+0000 7f3a484e9700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a401032b0 0x7f3a40197e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000ebc0 tx=0x7f3a3000eed0 comp rx=0 tx=0).stop 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.436+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 shutdown_connections 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.436+0000 7f3a484e9700 1 --2- 172.21.15.136:0/512331417 >> v2:172.21.15.136:6800/2365933975 conn(0x7f3a2c06d900 0x7f3a2c06fdc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.436+0000 7f3a484e9700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a40103c50 0x7f3a4019ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.436+0000 7f3a484e9700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a401020b0 0x7f3a401978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:46.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.436+0000 7f3a484e9700 1 --2- 172.21.15.136:0/512331417 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a401032b0 0x7f3a40197e10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:46.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.436+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 >> 172.21.15.136:0/512331417 conn(0x7f3a400fd690 msgr2=0x7f3a400ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:18:46.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.436+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 shutdown_connections 2024-03-26T05:18:46.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:46.436+0000 7f3a484e9700 1 -- 172.21.15.136:0/512331417 wait complete. 2024-03-26T05:18:46.765 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:46 smithi136 ceph-mon[29560]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T05:18:46.765 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:46.765 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:46.765 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:46 smithi136 ceph-mon[38106]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T05:18:46.765 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:46.765 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:46.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:46 smithi164 ceph-mon[33554]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T05:18:46.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:46.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:47.068 DEBUG:teuthology.orchestra.run.smithi136:osd.1> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T05:18:47.071 INFO:tasks.cephadm:Deploying osd.2 on smithi136 with /dev/vg_nvme/lv_2... 2024-03-26T05:18:47.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T05:18:47.358 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:47.769 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:47.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T05:18:47.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T05:18:47.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:18:47.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:47.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:47.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:47.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:18:47.771 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T05:18:47.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:47.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:47.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:48.073 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1[47991]: 2024-03-26T05:18:47.871+0000 7f74905e7700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T05:18:48.073 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:18:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1[47991]: 2024-03-26T05:18:47.873+0000 7f74883d2700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:18:49.060 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:48 smithi136 ceph-mon[29560]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' 2024-03-26T05:18:49.060 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:48 smithi136 ceph-mon[29560]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:49.060 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:49.060 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:48 smithi136 ceph-mon[38106]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' 2024-03-26T05:18:49.060 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:48 smithi136 ceph-mon[38106]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:49.060 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:49.198 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T05:18:49.199 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T05:18:49.199 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-26T05:18:49.199 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-26T05:18:49.199 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00940204 s, 1.1 GB/s 2024-03-26T05:18:49.199 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-26T05:18:49.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:48 smithi164 ceph-mon[33554]: from='osd.1 v2:172.21.15.136:6805/1223127568' entity='osd.1' 2024-03-26T05:18:49.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:48 smithi164 ceph-mon[33554]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T05:18:49.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:49.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[38106]: purged_snaps scrub starts 2024-03-26T05:18:49.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[38106]: purged_snaps scrub ok 2024-03-26T05:18:49.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[38106]: osd.1 v2:172.21.15.136:6805/1223127568 boot 2024-03-26T05:18:49.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[38106]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T05:18:49.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:49.976 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[29560]: purged_snaps scrub starts 2024-03-26T05:18:49.976 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[29560]: purged_snaps scrub ok 2024-03-26T05:18:49.977 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[29560]: osd.1 v2:172.21.15.136:6805/1223127568 boot 2024-03-26T05:18:49.977 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[29560]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T05:18:49.977 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:49 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:50.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:49 smithi164 ceph-mon[33554]: purged_snaps scrub starts 2024-03-26T05:18:50.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:49 smithi164 ceph-mon[33554]: purged_snaps scrub ok 2024-03-26T05:18:50.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:49 smithi164 ceph-mon[33554]: osd.1 v2:172.21.15.136:6805/1223127568 boot 2024-03-26T05:18:50.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:49 smithi164 ceph-mon[33554]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T05:18:50.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:49 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:18:50.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2024-03-26T05:18:50.973 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:50 smithi136 ceph-mon[38106]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T05:18:50.973 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:50 smithi136 ceph-mon[38106]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:50.973 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:50 smithi136 ceph-mon[29560]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T05:18:50.974 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:50 smithi136 ceph-mon[29560]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:51.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:50 smithi164 ceph-mon[33554]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T05:18:51.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:50 smithi164 ceph-mon[33554]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:51.269 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:18:53.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 -- 172.21.15.136:0/476519307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 msgr2=0x7f5f34104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:53.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 --2- 172.21.15.136:0/476519307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 0x7f5f34104f30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400b0a0 tx=0x7f5f2400b3b0 comp rx=0 tx=0).stop 2024-03-26T05:18:53.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 -- 172.21.15.136:0/476519307 shutdown_connections 2024-03-26T05:18:53.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 --2- 172.21.15.136:0/476519307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f34107da0 0x7f5f3410a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:53.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 --2- 172.21.15.136:0/476519307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 0x7f5f34104f30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:53.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 --2- 172.21.15.136:0/476519307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f34105470 0x7f5f34107860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:53.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 -- 172.21.15.136:0/476519307 >> 172.21.15.136:0/476519307 conn(0x7f5f340fc550 msgr2=0x7f5f340fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 -- 172.21.15.136:0/476519307 shutdown_connections 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.427+0000 7f5f390cf700 1 -- 172.21.15.136:0/476519307 wait complete. 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f390cf700 1 Processor -- start 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f390cf700 1 -- start start 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f390cf700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 0x7f5f34197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f390cf700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f34105470 0x7f5f34197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f390cf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f34107da0 0x7f5f3419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f390cf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3419f490 con 0x7f5f34107da0 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f390cf700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3419f5d0 con 0x7f5f34105470 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f390cf700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3419f760 con 0x7f5f34102b40 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f32d9d700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 0x7f5f34197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f3359e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f34107da0 0x7f5f3419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f32d9d700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 0x7f5f34197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:56918/0 (socket says 172.21.15.136:56918) 2024-03-26T05:18:53.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f32d9d700 1 -- 172.21.15.136:0/4266359186 learned_addr learned my addr 172.21.15.136:0/4266359186 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f2a59c700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f34105470 0x7f5f34197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f32d9d700 1 -- 172.21.15.136:0/4266359186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f34105470 msgr2=0x7f5f34197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f32d9d700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f34105470 0x7f5f34197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f32d9d700 1 -- 172.21.15.136:0/4266359186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f34107da0 msgr2=0x7f5f3419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f32d9d700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f34107da0 0x7f5f3419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f32d9d700 1 -- 172.21.15.136:0/4266359186 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c009520 con 0x7f5f34102b40 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.428+0000 7f5f3359e700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f34107da0 0x7f5f3419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.429+0000 7f5f32d9d700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 0x7f5f34197950 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5f24007a00 tx=0x7f5f24008fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.429+0000 7f5f30d99700 1 -- 172.21.15.136:0/4266359186 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2400e040 con 0x7f5f34102b40 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.429+0000 7f5f30d99700 1 -- 172.21.15.136:0/4266359186 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5f2400be70 con 0x7f5f34102b40 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.429+0000 7f5f30d99700 1 -- 172.21.15.136:0/4266359186 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f240052a0 con 0x7f5f34102b40 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.429+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f24009d00 con 0x7f5f34102b40 2024-03-26T05:18:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.429+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f34100bc0 con 0x7f5f34102b40 2024-03-26T05:18:53.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.430+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f340fe0d0 con 0x7f5f34102b40 2024-03-26T05:18:53.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.430+0000 7f5f30d99700 1 -- 172.21.15.136:0/4266359186 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 14) v1 ==== 91308+0+0 (secure 0 0 0) 0x7f5f2401d470 con 0x7f5f34102b40 2024-03-26T05:18:53.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.431+0000 7f5f30d99700 1 --2- 172.21.15.136:0/4266359186 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5f2006d900 0x7f5f2006fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:18:53.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.431+0000 7f5f30d99700 1 -- 172.21.15.136:0/4266359186 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 1833+0+0 (secure 0 0 0) 0x7f5f2408d5a0 con 0x7f5f34102b40 2024-03-26T05:18:53.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.431+0000 7f5f2a59c700 1 --2- 172.21.15.136:0/4266359186 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5f2006d900 0x7f5f2006fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:18:53.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.431+0000 7f5f2a59c700 1 --2- 172.21.15.136:0/4266359186 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5f2006d900 0x7f5f2006fdc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009df0 tx=0x7f5f1c012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:18:53.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.436+0000 7f5f30d99700 1 -- 172.21.15.136:0/4266359186 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5f240580c0 con 0x7f5f34102b40 2024-03-26T05:18:53.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:18:53.644+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f340624c0 con 0x7f5f2006d900 2024-03-26T05:18:53.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: Detected new or changed devices on smithi136 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi136 to 8032M 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:53.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: Detected new or changed devices on smithi136 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi136 to 8032M 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:53.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:53.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:53.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: Detected new or changed devices on smithi136 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi136 to 8032M 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:18:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:18:54.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:54 smithi136 ceph-mon[29560]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:54.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:54.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:54.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:54 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:54.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:54 smithi136 ceph-mon[38106]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:54.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:54.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:54.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:54 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:54.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:54 smithi164 ceph-mon[33554]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:18:54.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:18:54.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:18:54.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:54 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:18:55.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:55 smithi136 ceph-mon[29560]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:55.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:55 smithi136 ceph-mon[38106]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:55.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:55 smithi164 ceph-mon[33554]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:56.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1312154401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]: dispatch 2024-03-26T05:18:56.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]: dispatch 2024-03-26T05:18:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]': finished 2024-03-26T05:18:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[38106]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T05:18:56.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:18:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1312154401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]: dispatch 2024-03-26T05:18:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]: dispatch 2024-03-26T05:18:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]': finished 2024-03-26T05:18:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[29560]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T05:18:56.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:18:56.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1312154401' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]: dispatch 2024-03-26T05:18:56.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]: dispatch 2024-03-26T05:18:56.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e60acb1-d576-40cf-9b67-28f11a3a2846"}]': finished 2024-03-26T05:18:56.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:56 smithi164 ceph-mon[33554]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T05:18:56.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:18:57.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:57 smithi136 ceph-mon[29560]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:57.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4271030620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:57.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:57 smithi136 ceph-mon[38106]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:57.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4271030620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:57.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:57 smithi164 ceph-mon[33554]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:57.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4271030620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:18:59.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:18:59 smithi136 ceph-mon[29560]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:59.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:18:59 smithi136 ceph-mon[38106]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:18:59.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:18:59 smithi164 ceph-mon[33554]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:01.775 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:01 smithi136 ceph-mon[38106]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:01.776 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:01 smithi136 ceph-mon[29560]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:01.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:01 smithi164 ceph-mon[33554]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:03.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:03 smithi136 ceph-mon[38106]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:03.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:03 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T05:19:03.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:03 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:03.780 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:03 smithi136 ceph-mon[29560]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:03.780 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:03 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T05:19:03.780 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:03 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:03.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:03 smithi164 ceph-mon[33554]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:03.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:03 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T05:19:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:03 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:04.811 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:04 smithi136 ceph-mon[38106]: Deploying daemon osd.2 on smithi136 2024-03-26T05:19:04.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:04 smithi136 ceph-mon[29560]: Deploying daemon osd.2 on smithi136 2024-03-26T05:19:04.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:04 smithi164 ceph-mon[33554]: Deploying daemon osd.2 on smithi136 2024-03-26T05:19:05.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:05 smithi136 ceph-mon[29560]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:05 smithi136 ceph-mon[38106]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:05.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:05 smithi164 ceph-mon[33554]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:07.759 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:07 smithi136 ceph-mon[29560]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:07.759 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:07 smithi136 ceph-mon[38106]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:07.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:07 smithi164 ceph-mon[33554]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:09.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:09 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:09.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:09 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:09.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:09 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:09.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:09 smithi136 ceph-mon[29560]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:09.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:09 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:09.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:09 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:09.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:09 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:09.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:09 smithi136 ceph-mon[38106]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:09.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:09 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:09.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:09 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:09.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:09 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:09.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:09 smithi164 ceph-mon[33554]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:11.719 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:11 smithi136 ceph-mon[38106]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:11.719 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:11 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:11.719 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:11 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:11.719 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:11 smithi136 ceph-mon[38106]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T05:19:11.719 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:11 smithi136 ceph-mon[29560]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:11.719 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:11 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:11.719 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:11 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:11.719 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:11 smithi136 ceph-mon[29560]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T05:19:11.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:11 smithi164 ceph-mon[33554]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:11.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:11 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:11.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:11 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:11.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:11 smithi164 ceph-mon[33554]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T05:19:12.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.076+0000 7f5f30d99700 1 -- 172.21.15.136:0/4266359186 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5f340624c0 con 0x7f5f2006d900 2024-03-26T05:19:12.067 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 2 on host 'smithi136' 2024-03-26T05:19:12.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5f2006d900 msgr2=0x7f5f2006fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:12.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 --2- 172.21.15.136:0/4266359186 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5f2006d900 0x7f5f2006fdc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009df0 tx=0x7f5f1c012040 comp rx=0 tx=0).stop 2024-03-26T05:19:12.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 msgr2=0x7f5f34197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:12.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 0x7f5f34197950 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5f24007a00 tx=0x7f5f24008fe0 comp rx=0 tx=0).stop 2024-03-26T05:19:12.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 shutdown_connections 2024-03-26T05:19:12.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 --2- 172.21.15.136:0/4266359186 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5f2006d900 0x7f5f2006fdc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:12.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f34107da0 0x7f5f3419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:12.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f34102b40 0x7f5f34197950 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:12.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 --2- 172.21.15.136:0/4266359186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f34105470 0x7f5f34197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:12.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 >> 172.21.15.136:0/4266359186 conn(0x7f5f340fc550 msgr2=0x7f5f341037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:19:12.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 shutdown_connections 2024-03-26T05:19:12.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:12.079+0000 7f5f390cf700 1 -- 172.21.15.136:0/4266359186 wait complete. 2024-03-26T05:19:12.744 DEBUG:teuthology.orchestra.run.smithi136:osd.2> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T05:19:12.747 INFO:tasks.cephadm:Deploying osd.3 on smithi136 with /dev/vg_nvme/lv_1... 2024-03-26T05:19:12.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T05:19:13.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:13.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:13.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[29560]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T05:19:13.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[29560]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T05:19:13.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[29560]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:13.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:13.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[29560]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[38106]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T05:19:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[38106]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T05:19:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[38106]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-mon[38106]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:13.325 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2[53595]: 2024-03-26T05:19:13.264+0000 7f8ee389d700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T05:19:13.325 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:13 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2[53595]: 2024-03-26T05:19:13.267+0000 7f8edf690700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:19:13.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:13 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:13 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:13 smithi164 ceph-mon[33554]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T05:19:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:13 smithi164 ceph-mon[33554]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T05:19:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:13 smithi164 ceph-mon[33554]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:13 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:13 smithi164 ceph-mon[33554]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:13.970 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:14.491 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:14 smithi136 ceph-mon[38106]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:19:14.491 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:14 smithi136 ceph-mon[38106]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T05:19:14.491 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:14 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:14.491 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:14 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:14.492 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:14 smithi136 ceph-mon[29560]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:19:14.492 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:14 smithi136 ceph-mon[29560]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T05:19:14.492 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:14 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:14.492 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:14 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:14.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:14 smithi164 ceph-mon[33554]: from='osd.2 v2:172.21.15.136:6809/1411885928' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:19:14.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:14 smithi164 ceph-mon[33554]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T05:19:14.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:14 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:14.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:14 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:15.563 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[29560]: purged_snaps scrub starts 2024-03-26T05:19:15.563 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[29560]: purged_snaps scrub ok 2024-03-26T05:19:15.563 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[29560]: osd.2 v2:172.21.15.136:6809/1411885928 boot 2024-03-26T05:19:15.563 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[29560]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[29560]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[38106]: purged_snaps scrub starts 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[38106]: purged_snaps scrub ok 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[38106]: osd.2 v2:172.21.15.136:6809/1411885928 boot 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[38106]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[38106]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:15.564 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:19:15.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:15 smithi164 ceph-mon[33554]: purged_snaps scrub starts 2024-03-26T05:19:15.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:15 smithi164 ceph-mon[33554]: purged_snaps scrub ok 2024-03-26T05:19:15.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:15 smithi164 ceph-mon[33554]: osd.2 v2:172.21.15.136:6809/1411885928 boot 2024-03-26T05:19:15.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:15 smithi164 ceph-mon[33554]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T05:19:15.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:19:15.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:15 smithi164 ceph-mon[33554]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T05:19:15.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:19:16.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:16.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.452 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:16.452 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:16.452 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:17.073 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T05:19:17.073 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T05:19:17.073 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-03-26T05:19:17.073 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-03-26T05:19:17.073 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00783153 s, 1.3 GB/s 2024-03-26T05:19:17.073 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-03-26T05:19:17.240 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:19:16 smithi136 sudo[57405]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T05:19:17.241 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:19:16 smithi136 sudo[57405]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:17.241 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:19:16 smithi136 sudo[57405]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:17.241 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57405]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:17.512 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57502]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T05:19:17.512 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57502]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:17.512 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57502]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:17.512 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57502]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: Detected new or changed devices on smithi136 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi136 to 5354M 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:17.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: Detected new or changed devices on smithi136 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi136 to 5354M 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:17.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:17.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: Detected new or changed devices on smithi136 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi136 to 5354M 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:17.768 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57509]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T05:19:17.769 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57509]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:17.769 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57509]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:17.769 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57509]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:17.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2024-03-26T05:19:18.026 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57512]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T05:19:18.026 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57512]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:18.026 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57512]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:18.026 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:19:17 smithi136 sudo[57512]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:18.054 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:19:18.282 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57557]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T05:19:18.282 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57557]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:18.282 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57557]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:18.282 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57557]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:18.543 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57594]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T05:19:18.544 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57594]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:18.544 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57594]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:18.544 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57594]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:18.799 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57611]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T05:19:18.800 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57611]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:18.800 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57611]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:18.800 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57611]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:19.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:18 smithi164 ceph-mon[33554]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T05:19:19.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:18 smithi164 ceph-mon[33554]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T05:19:19.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:18 smithi164 ceph-mon[33554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:19.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 sudo[36142]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T05:19:19.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 sudo[36142]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:19.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 sudo[36142]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:19.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 sudo[36142]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:19.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:18 smithi136 ceph-mon[38106]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T05:19:19.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:18 smithi136 ceph-mon[38106]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T05:19:19.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:18 smithi136 ceph-mon[38106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:19.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57617]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T05:19:19.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57617]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:19.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57617]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:19.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 sudo[57617]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:19.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 ceph-mon[29560]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T05:19:19.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 ceph-mon[29560]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T05:19:19.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:18 smithi136 ceph-mon[29560]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:19.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 sudo[36145]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T05:19:19.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 sudo[36145]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:19.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 sudo[36145]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 sudo[36145]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:19.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 -- 172.21.15.136:0/2506051520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401020f0 msgr2=0x7f5a40102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 --2- 172.21.15.136:0/2506051520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401020f0 0x7f5a40102510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0099c0 tx=0x7f5a3c009cd0 comp rx=0 tx=0).stop 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 -- 172.21.15.136:0/2506051520 shutdown_connections 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 --2- 172.21.15.136:0/2506051520 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5a40103c90 0x7f5a4010a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 --2- 172.21.15.136:0/2506051520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401020f0 0x7f5a40102510 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 --2- 172.21.15.136:0/2506051520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a401032f0 0x7f5a40103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 -- 172.21.15.136:0/2506051520 >> 172.21.15.136:0/2506051520 conn(0x7f5a400fd6d0 msgr2=0x7f5a400ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 -- 172.21.15.136:0/2506051520 shutdown_connections 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.714+0000 7f5a4753a700 1 -- 172.21.15.136:0/2506051520 wait complete. 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a4753a700 1 Processor -- start 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a4753a700 1 -- start start 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a4753a700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a401020f0 0x7f5a401977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a4753a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401032f0 0x7f5a40197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a4753a700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5a40103c90 0x7f5a4019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:19:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a4753a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019f350 con 0x7f5a401032f0 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a4753a700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019f490 con 0x7f5a401020f0 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a4753a700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019f620 con 0x7f5a40103c90 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a44ad5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401032f0 0x7f5a40197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a45ad7700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5a40103c90 0x7f5a4019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a44ad5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401032f0 0x7f5a40197d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:42210/0 (socket says 172.21.15.136:42210) 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a44ad5700 1 -- 172.21.15.136:0/1874480338 learned_addr learned my addr 172.21.15.136:0/1874480338 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.715+0000 7f5a452d6700 1 --2- 172.21.15.136:0/1874480338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a401020f0 0x7f5a401977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a44ad5700 1 -- 172.21.15.136:0/1874480338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5a40103c90 msgr2=0x7f5a4019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a44ad5700 1 --2- 172.21.15.136:0/1874480338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5a40103c90 0x7f5a4019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a44ad5700 1 -- 172.21.15.136:0/1874480338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a401020f0 msgr2=0x7f5a401977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a44ad5700 1 --2- 172.21.15.136:0/1874480338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a401020f0 0x7f5a401977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a44ad5700 1 -- 172.21.15.136:0/1874480338 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c0095f0 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a44ad5700 1 --2- 172.21.15.136:0/1874480338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401032f0 0x7f5a40197d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000e9d0 tx=0x7f5a3000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a367fc700 1 -- 172.21.15.136:0/1874480338 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30009790 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a367fc700 1 -- 172.21.15.136:0/1874480338 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5a3000ee70 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a40074310 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a367fc700 1 -- 172.21.15.136:0/1874480338 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30018680 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.716+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a40074810 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.717+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a401094b0 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.718+0000 7f5a367fc700 1 -- 172.21.15.136:0/1874480338 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f5a30010430 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.718+0000 7f5a367fc700 1 --2- 172.21.15.136:0/1874480338 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5a2c06d990 0x7f5a2c06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.718+0000 7f5a367fc700 1 -- 172.21.15.136:0/1874480338 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 2511+0+0 (secure 0 0 0) 0x7f5a30014070 con 0x7f5a401032f0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.718+0000 7f5a452d6700 1 --2- 172.21.15.136:0/1874480338 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5a2c06d990 0x7f5a2c06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:19:19.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.718+0000 7f5a452d6700 1 --2- 172.21.15.136:0/1874480338 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5a2c06d990 0x7f5a2c06fe50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c000c00 tx=0x7f5a3c01d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:19:19.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.723+0000 7f5a367fc700 1 -- 172.21.15.136:0/1874480338 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5a30053280 con 0x7f5a401032f0 2024-03-26T05:19:19.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 sudo[57646]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T05:19:19.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 sudo[57646]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:19.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 sudo[57646]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:19.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 sudo[57646]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:19.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:19.938+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a4010d490 con 0x7f5a2c06d990 2024-03-26T05:19:20.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:20.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:20.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:20.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:20.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:20.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 sudo[57671]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 sudo[57671]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 sudo[57671]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 sudo[57671]: pam_unix(sudo:session): session closed for user root 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:19:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:19:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:19:21.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[29560]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[29560]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[29560]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[29560]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[38106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[38106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[38106]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:19:21.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:19:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:19:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:20 smithi136 ceph-mon[38106]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:21.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:20 smithi164 ceph-mon[33554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T05:19:21.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:20 smithi164 ceph-mon[33554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T05:19:21.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:20 smithi164 ceph-mon[33554]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:19:21.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:20 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:19:21.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:20 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:19:21.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:20 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:21.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:20 smithi164 ceph-mon[33554]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[29560]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4084237442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]: dispatch 2024-03-26T05:19:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]: dispatch 2024-03-26T05:19:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]': finished 2024-03-26T05:19:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[29560]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T05:19:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2918197152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:19:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[38106]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4084237442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]: dispatch 2024-03-26T05:19:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]: dispatch 2024-03-26T05:19:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]': finished 2024-03-26T05:19:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[38106]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T05:19:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2918197152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:19:23.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:23 smithi164 ceph-mon[33554]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:23.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4084237442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]: dispatch 2024-03-26T05:19:23.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]: dispatch 2024-03-26T05:19:23.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c52d96a6-a66d-446b-aa35-891cc84fa99b"}]': finished 2024-03-26T05:19:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:23 smithi164 ceph-mon[33554]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T05:19:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:23 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2918197152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:19:25.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:25 smithi136 ceph-mon[29560]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:25.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:25 smithi136 ceph-mon[38106]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:25.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:25 smithi164 ceph-mon[33554]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:27.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:27 smithi136 ceph-mon[38106]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-26T05:19:27.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:27 smithi136 ceph-mon[29560]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-26T05:19:27.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:27 smithi164 ceph-mon[33554]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-26T05:19:29.858 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:29 smithi136 ceph-mon[29560]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-26T05:19:29.858 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:29 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T05:19:29.858 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:29 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:29.859 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:29 smithi136 ceph-mon[38106]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-26T05:19:29.859 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:29 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T05:19:29.859 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:29 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:29.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:29 smithi164 ceph-mon[33554]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-26T05:19:29.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:29 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T05:19:29.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:29 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:30.955 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:30 smithi136 ceph-mon[29560]: Deploying daemon osd.3 on smithi136 2024-03-26T05:19:30.956 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:30 smithi136 ceph-mon[38106]: Deploying daemon osd.3 on smithi136 2024-03-26T05:19:30.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:30 smithi164 ceph-mon[33554]: Deploying daemon osd.3 on smithi136 2024-03-26T05:19:31.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:31 smithi136 ceph-mon[29560]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T05:19:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:31 smithi136 ceph-mon[38106]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T05:19:31.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:31 smithi164 ceph-mon[33554]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T05:19:33.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:33 smithi136 ceph-mon[29560]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T05:19:33.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:33 smithi136 ceph-mon[38106]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T05:19:33.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:33 smithi164 ceph-mon[33554]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T05:19:35.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:35 smithi136 ceph-mon[29560]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 779 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T05:19:35.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:35 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:35.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:35 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:35.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:35 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:35.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:35 smithi136 ceph-mon[38106]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 779 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T05:19:35.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:35 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:35.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:35 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:35.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:35 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:35 smithi164 ceph-mon[33554]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 779 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T05:19:35.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:35 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:35.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:35 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:35.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:35 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:37.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:37 smithi136 ceph-mon[29560]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-26T05:19:37.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:37 smithi136 ceph-mon[29560]: from='osd.3 v2:172.21.15.136:6813/802386900' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T05:19:37.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:37 smithi136 ceph-mon[29560]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T05:19:37.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:37 smithi136 ceph-mon[38106]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-26T05:19:37.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:37 smithi136 ceph-mon[38106]: from='osd.3 v2:172.21.15.136:6813/802386900' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T05:19:37.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:37 smithi136 ceph-mon[38106]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T05:19:37.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:37 smithi164 ceph-mon[33554]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-26T05:19:37.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:37 smithi164 ceph-mon[33554]: from='osd.3 v2:172.21.15.136:6813/802386900' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T05:19:37.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:37 smithi164 ceph-mon[33554]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T05:19:38.105 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 3 on host 'smithi136' 2024-03-26T05:19:38.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.115+0000 7f5a367fc700 1 -- 172.21.15.136:0/1874480338 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5a4010d490 con 0x7f5a2c06d990 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5a2c06d990 msgr2=0x7f5a2c06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 --2- 172.21.15.136:0/1874480338 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5a2c06d990 0x7f5a2c06fe50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c000c00 tx=0x7f5a3c01d040 comp rx=0 tx=0).stop 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401032f0 msgr2=0x7f5a40197d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 --2- 172.21.15.136:0/1874480338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401032f0 0x7f5a40197d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000e9d0 tx=0x7f5a3000ece0 comp rx=0 tx=0).stop 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 shutdown_connections 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 --2- 172.21.15.136:0/1874480338 >> v2:172.21.15.136:6800/2365933975 conn(0x7f5a2c06d990 0x7f5a2c06fe50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 --2- 172.21.15.136:0/1874480338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a401020f0 0x7f5a401977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 --2- 172.21.15.136:0/1874480338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5a40103c90 0x7f5a4019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 --2- 172.21.15.136:0/1874480338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a401032f0 0x7f5a40197d00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:38.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 >> 172.21.15.136:0/1874480338 conn(0x7f5a400fd6d0 msgr2=0x7f5a401065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:19:38.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 shutdown_connections 2024-03-26T05:19:38.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:19:38.118+0000 7f5a4753a700 1 -- 172.21.15.136:0/1874480338 wait complete. 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[38106]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[38106]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[38106]: from='osd.3 v2:172.21.15.136:6813/802386900' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[38106]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[29560]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T05:19:38.703 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[29560]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T05:19:38.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:38.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[29560]: from='osd.3 v2:172.21.15.136:6813/802386900' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:38.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[29560]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:38.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:38.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:38.732 DEBUG:teuthology.orchestra.run.smithi136:osd.3> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T05:19:38.734 INFO:tasks.cephadm:Deploying osd.4 on smithi164 with /dev/vg_nvme/lv_4... 2024-03-26T05:19:38.734 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T05:19:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:38 smithi164 ceph-mon[33554]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T05:19:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:38 smithi164 ceph-mon[33554]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T05:19:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:38 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:38 smithi164 ceph-mon[33554]: from='osd.3 v2:172.21.15.136:6813/802386900' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:38 smithi164 ceph-mon[33554]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-03-26T05:19:38.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:38 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:38 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:38.979 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:19:39.073 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3[59564]: 2024-03-26T05:19:38.832+0000 7f64d1f69700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T05:19:39.073 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:19:38 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3[59564]: 2024-03-26T05:19:38.837+0000 7f64c9d54700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:19:39.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:39 smithi164 ceph-mon[33554]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:39.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:39 smithi164 ceph-mon[33554]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:19:39.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:39 smithi164 ceph-mon[33554]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T05:19:39.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:39.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:39.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:39 smithi164 ceph-mon[33554]: from='osd.3 ' entity='osd.3' 2024-03-26T05:19:39.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:39.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:39.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[38106]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:39.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[38106]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[38106]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[38106]: from='osd.3 ' entity='osd.3' 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[29560]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[29560]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[29560]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T05:19:39.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[29560]: from='osd.3 ' entity='osd.3' 2024-03-26T05:19:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:40.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:40 smithi164 ceph-mon[33554]: purged_snaps scrub starts 2024-03-26T05:19:40.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:40 smithi164 ceph-mon[33554]: purged_snaps scrub ok 2024-03-26T05:19:40.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:40 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:40.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:40 smithi164 ceph-mon[33554]: osd.3 v2:172.21.15.136:6813/802386900 boot 2024-03-26T05:19:40.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:40 smithi164 ceph-mon[33554]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T05:19:40.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:40 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:40.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[38106]: purged_snaps scrub starts 2024-03-26T05:19:40.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[38106]: purged_snaps scrub ok 2024-03-26T05:19:40.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:40.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[38106]: osd.3 v2:172.21.15.136:6813/802386900 boot 2024-03-26T05:19:40.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[38106]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T05:19:40.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:40.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[29560]: purged_snaps scrub starts 2024-03-26T05:19:40.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[29560]: purged_snaps scrub ok 2024-03-26T05:19:40.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:40.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[29560]: osd.3 v2:172.21.15.136:6813/802386900 boot 2024-03-26T05:19:40.804 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[29560]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T05:19:40.804 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:40 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:19:40.930 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T05:19:40.930 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T05:19:40.930 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-26T05:19:40.930 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-26T05:19:40.931 INFO:teuthology.orchestra.run.smithi164.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131792 s, 796 MB/s 2024-03-26T05:19:40.931 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-26T05:19:41.494 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_4 2024-03-26T05:19:41.737 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:19:41.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:41 smithi164 ceph-mon[33554]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:41.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:41 smithi164 ceph-mon[33554]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T05:19:41.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:41 smithi136 ceph-mon[29560]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:41.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:41 smithi136 ceph-mon[29560]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T05:19:41.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:41 smithi136 ceph-mon[38106]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T05:19:41.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:41 smithi136 ceph-mon[38106]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T05:19:43.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 -- 172.21.15.164:0/2378051790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a34102b80 msgr2=0x7f4a34109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/2378051790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a34102b80 0x7f4a34109430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00b0d0 tx=0x7f4a2c00b3e0 comp rx=0 tx=0).stop 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 -- 172.21.15.164:0/2378051790 shutdown_connections 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/2378051790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a34102b80 0x7f4a34109430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/2378051790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a34100fe0 0x7f4a34101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/2378051790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a341021e0 0x7f4a34102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 -- 172.21.15.164:0/2378051790 >> 172.21.15.164:0/2378051790 conn(0x7f4a340fc5a0 msgr2=0x7f4a340fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 -- 172.21.15.164:0/2378051790 shutdown_connections 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 -- 172.21.15.164:0/2378051790 wait complete. 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 Processor -- start 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.414+0000 7f4a3af7f700 1 -- start start 2024-03-26T05:19:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a3af7f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a34100fe0 0x7f4a3410b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a3af7f700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a341021e0 0x7f4a34109f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a3af7f700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a34102b80 0x7f4a3410a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a3af7f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3410ac00 con 0x7f4a34100fe0 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a3af7f700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3410ad70 con 0x7f4a341021e0 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a3af7f700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a341a18b0 con 0x7f4a34102b80 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a341021e0 0x7f4a34109f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a341021e0 0x7f4a34109f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:41608/0 (socket says 172.21.15.164:41608) 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 -- 172.21.15.164:0/3559642932 learned_addr learned my addr 172.21.15.164:0/3559642932 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a38d1b700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a34100fe0 0x7f4a3410b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a3951c700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a34102b80 0x7f4a3410a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 -- 172.21.15.164:0/3559642932 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a34102b80 msgr2=0x7f4a3410a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a34102b80 0x7f4a3410a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 -- 172.21.15.164:0/3559642932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a34100fe0 msgr2=0x7f4a3410b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a34100fe0 0x7f4a3410b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:19:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 -- 172.21.15.164:0/3559642932 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c009d00 con 0x7f4a341021e0 2024-03-26T05:19:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a33fff700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a341021e0 0x7f4a34109f40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000e9a0 tx=0x7f4a2000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:19:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a31ffb700 1 -- 172.21.15.164:0/3559642932 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20009790 con 0x7f4a341021e0 2024-03-26T05:19:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a341a1b90 con 0x7f4a341021e0 2024-03-26T05:19:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a31ffb700 1 -- 172.21.15.164:0/3559642932 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f4a20010430 con 0x7f4a341021e0 2024-03-26T05:19:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.415+0000 7f4a31ffb700 1 -- 172.21.15.164:0/3559642932 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20018680 con 0x7f4a341021e0 2024-03-26T05:19:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.416+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a341a2090 con 0x7f4a341021e0 2024-03-26T05:19:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.417+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a34108340 con 0x7f4a341021e0 2024-03-26T05:19:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.417+0000 7f4a31ffb700 1 -- 172.21.15.164:0/3559642932 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f4a2000ca10 con 0x7f4a341021e0 2024-03-26T05:19:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.418+0000 7f4a31ffb700 1 --2- 172.21.15.164:0/3559642932 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4a2406d990 0x7f4a2406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:19:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.418+0000 7f4a31ffb700 1 -- 172.21.15.164:0/3559642932 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 2803+0+0 (secure 0 0 0) 0x7f4a20014070 con 0x7f4a341021e0 2024-03-26T05:19:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.418+0000 7f4a38d1b700 1 --2- 172.21.15.164:0/3559642932 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4a2406d990 0x7f4a2406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:19:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.418+0000 7f4a38d1b700 1 --2- 172.21.15.164:0/3559642932 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4a2406d990 0x7f4a2406fe50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4a34102040 tx=0x7f4a28007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:19:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.422+0000 7f4a31ffb700 1 -- 172.21.15.164:0/3559642932 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4a20053280 con 0x7f4a341021e0 2024-03-26T05:19:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:19:43.628+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a340624c0 con 0x7f4a2406d990 2024-03-26T05:19:43.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:43.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: Detected new or changed devices on smithi136 2024-03-26T05:19:43.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi136 to 4016M 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:43 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: Detected new or changed devices on smithi136 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi136 to 4016M 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:44.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: Detected new or changed devices on smithi136 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:19:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi136 to 4016M 2024-03-26T05:19:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:19:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:43 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:44.806 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:44 smithi164 ceph-mon[33554]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:19:44.806 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:44 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:19:44.806 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:44 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:19:44.807 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:44 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:45.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:44 smithi136 ceph-mon[29560]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:19:45.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:44 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:19:45.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:44 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:19:45.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:44 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:45.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:44 smithi136 ceph-mon[38106]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:19:45.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:44 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:19:45.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:44 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:19:45.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:44 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:45.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:45 smithi164 ceph-mon[33554]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:46.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:45 smithi136 ceph-mon[29560]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:46.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:45 smithi136 ceph-mon[38106]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:46.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/2070087617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]: dispatch 2024-03-26T05:19:46.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]: dispatch 2024-03-26T05:19:46.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]': finished 2024-03-26T05:19:46.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:46 smithi164 ceph-mon[33554]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T05:19:46.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:46 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:19:47.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/2070087617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]: dispatch 2024-03-26T05:19:47.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]: dispatch 2024-03-26T05:19:47.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]': finished 2024-03-26T05:19:47.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[29560]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T05:19:47.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:19:47.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/2070087617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]: dispatch 2024-03-26T05:19:47.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]: dispatch 2024-03-26T05:19:47.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a304c9c4-c2c8-4e3d-8292-48bc13c48ffd"}]': finished 2024-03-26T05:19:47.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[38106]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T05:19:47.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:46 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:19:47.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:47 smithi164 ceph-mon[33554]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:47.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/1447571737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:19:48.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:47 smithi136 ceph-mon[29560]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:48.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/1447571737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:19:48.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:47 smithi136 ceph-mon[38106]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:48.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/1447571737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:19:49.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:49 smithi164 ceph-mon[33554]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:50.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:49 smithi136 ceph-mon[38106]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:50.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:49 smithi136 ceph-mon[29560]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:51.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:51 smithi164 ceph-mon[33554]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:51.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:51 smithi136 ceph-mon[38106]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:51.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:51 smithi136 ceph-mon[29560]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:52.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:52 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T05:19:52.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:52 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:53.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:52 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T05:19:53.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:52 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:53.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:52 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T05:19:53.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:52 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:19:53.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:53 smithi164 ceph-mon[33554]: Deploying daemon osd.4 on smithi164 2024-03-26T05:19:53.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:53 smithi164 ceph-mon[33554]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:54.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:53 smithi136 ceph-mon[29560]: Deploying daemon osd.4 on smithi164 2024-03-26T05:19:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:53 smithi136 ceph-mon[29560]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:53 smithi136 ceph-mon[38106]: Deploying daemon osd.4 on smithi164 2024-03-26T05:19:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:53 smithi136 ceph-mon[38106]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:55.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:55 smithi164 ceph-mon[33554]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:56.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:55 smithi136 ceph-mon[29560]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:56.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:55 smithi136 ceph-mon[38106]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:57.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:57 smithi164 ceph-mon[33554]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:57.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:57.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:57.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:58.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:57 smithi136 ceph-mon[29560]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:58.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:58.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:58.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:58.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:57 smithi136 ceph-mon[38106]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:19:58.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:58.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:19:58.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:19:59.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:19:59 smithi164 ceph-mon[33554]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:00.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:19:59 smithi136 ceph-mon[29560]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:00.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:19:59 smithi136 ceph-mon[38106]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:00.277 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 4 on host 'smithi164' 2024-03-26T05:20:00.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.286+0000 7f4a31ffb700 1 -- 172.21.15.164:0/3559642932 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4a340624c0 con 0x7f4a2406d990 2024-03-26T05:20:00.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4a2406d990 msgr2=0x7f4a2406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/3559642932 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4a2406d990 0x7f4a2406fe50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4a34102040 tx=0x7f4a28007400 comp rx=0 tx=0).stop 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a341021e0 msgr2=0x7f4a34109f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a341021e0 0x7f4a34109f40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000e9a0 tx=0x7f4a2000ecb0 comp rx=0 tx=0).stop 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 shutdown_connections 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/3559642932 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4a2406d990 0x7f4a2406fe50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a34102b80 0x7f4a3410a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a34100fe0 0x7f4a3410b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 --2- 172.21.15.164:0/3559642932 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a341021e0 0x7f4a34109f40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 >> 172.21.15.164:0/3559642932 conn(0x7f4a340fc5a0 msgr2=0x7f4a34105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 shutdown_connections 2024-03-26T05:20:00.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:00.289+0000 7f4a3af7f700 1 -- 172.21.15.164:0/3559642932 wait complete. 2024-03-26T05:20:00.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:00 smithi164 ceph-mon[33554]: overall HEALTH_OK 2024-03-26T05:20:00.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:00.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:00 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:00.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:00 smithi164 ceph-mon[33554]: from='osd.4 v2:172.21.15.164:6800/952209544' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:20:00.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:00 smithi164 ceph-mon[33554]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:20:01.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[29560]: overall HEALTH_OK 2024-03-26T05:20:01.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:01.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:01.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[29560]: from='osd.4 v2:172.21.15.164:6800/952209544' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:20:01.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[29560]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:20:01.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[38106]: overall HEALTH_OK 2024-03-26T05:20:01.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:01.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:01.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[38106]: from='osd.4 v2:172.21.15.164:6800/952209544' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:20:01.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:00 smithi136 ceph-mon[38106]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T05:20:01.114 DEBUG:teuthology.orchestra.run.smithi164:osd.4> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T05:20:01.117 INFO:tasks.cephadm:Deploying osd.5 on smithi164 with /dev/vg_nvme/lv_3... 2024-03-26T05:20:01.117 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T05:20:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:01.855 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:01 smithi164 ceph-mon[33554]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:01.855 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:01 smithi164 ceph-mon[33554]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T05:20:01.855 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:01 smithi164 ceph-mon[33554]: from='osd.4 v2:172.21.15.164:6800/952209544' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:01.855 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:01 smithi164 ceph-mon[33554]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T05:20:01.856 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:01.856 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:01 smithi164 ceph-mon[33554]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:01.856 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:01.856 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:02.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[29560]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:02.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[29560]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[29560]: from='osd.4 v2:172.21.15.164:6800/952209544' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[29560]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[29560]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[38106]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[38106]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[38106]: from='osd.4 v2:172.21.15.164:6800/952209544' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:02.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[38106]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T05:20:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[38106]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:02.556 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:20:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:20:02.259+0000 7fd55a4ed700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T05:20:02.556 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:20:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:20:02.263+0000 7fd5562e0700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:20:03.301 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T05:20:03.301 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T05:20:03.301 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-26T05:20:03.301 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-26T05:20:03.301 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127596 s, 822 MB/s 2024-03-26T05:20:03.301 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-26T05:20:03.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:03 smithi164 ceph-mon[33554]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:03.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:03 smithi164 ceph-mon[33554]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T05:20:03.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:03 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:03.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:03 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:03.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:03 smithi164 ceph-mon[33554]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:03.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[38106]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:03.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[38106]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T05:20:03.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:03.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:03.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[38106]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:03.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[29560]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:03.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[29560]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T05:20:03.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:03.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:03.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:03 smithi136 ceph-mon[29560]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T05:20:04.162 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_3 2024-03-26T05:20:04.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:04 smithi164 ceph-mon[33554]: purged_snaps scrub starts 2024-03-26T05:20:04.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:04 smithi164 ceph-mon[33554]: purged_snaps scrub ok 2024-03-26T05:20:04.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:04 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:04.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:04 smithi164 ceph-mon[33554]: osd.4 v2:172.21.15.164:6800/952209544 boot 2024-03-26T05:20:04.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:04 smithi164 ceph-mon[33554]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T05:20:04.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:04 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:04.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:04 smithi164 ceph-mon[33554]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T05:20:04.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[29560]: purged_snaps scrub starts 2024-03-26T05:20:04.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[29560]: purged_snaps scrub ok 2024-03-26T05:20:04.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:04.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[29560]: osd.4 v2:172.21.15.164:6800/952209544 boot 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[29560]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[29560]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[38106]: purged_snaps scrub starts 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[38106]: purged_snaps scrub ok 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[38106]: osd.4 v2:172.21.15.164:6800/952209544 boot 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[38106]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:20:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:04 smithi136 ceph-mon[38106]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T05:20:04.596 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:05.386 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:05 smithi164 ceph-mon[33554]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:05.386 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:05 smithi164 ceph-mon[33554]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T05:20:05.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:05 smithi136 ceph-mon[29560]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:05.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:05 smithi136 ceph-mon[29560]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T05:20:05.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:05 smithi136 ceph-mon[38106]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:05.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:05 smithi136 ceph-mon[38106]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T05:20:06.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T05:20:06.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: Detected new or changed devices on smithi164 2024-03-26T05:20:06.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:06.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:06.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:06.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi164 to 17088M 2024-03-26T05:20:06.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:06.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:06.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:06.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:06.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:06.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:06.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:06.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:06.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:06 smithi164 ceph-mon[33554]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:07.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T05:20:07.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: Detected new or changed devices on smithi164 2024-03-26T05:20:07.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi164 to 17088M 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[29560]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: Detected new or changed devices on smithi164 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi164 to 17088M 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:07.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:07.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:06 smithi136 ceph-mon[38106]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 -- 172.21.15.164:0/2455858581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9330100fe0 msgr2=0x7f9330101400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 --2- 172.21.15.164:0/2455858581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9330100fe0 0x7f9330101400 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f93200099c0 tx=0x7f9320009cd0 comp rx=0 tx=0).stop 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 -- 172.21.15.164:0/2455858581 shutdown_connections 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 --2- 172.21.15.164:0/2455858581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9330100fe0 0x7f9330101400 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 --2- 172.21.15.164:0/2455858581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9330102b80 0x7f9330109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 --2- 172.21.15.164:0/2455858581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93301021e0 0x7f9330102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 -- 172.21.15.164:0/2455858581 >> 172.21.15.164:0/2455858581 conn(0x7f93300fc5a0 msgr2=0x7f93300fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 -- 172.21.15.164:0/2455858581 shutdown_connections 2024-03-26T05:20:07.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 -- 172.21.15.164:0/2455858581 wait complete. 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 Processor -- start 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.154+0000 7f9335899700 1 -- start start 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9335899700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9330100fe0 0x7f9330104f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9335899700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93301021e0 0x7f93301035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9335899700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9330102b80 0x7f9330103b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9335899700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9330104280 con 0x7f9330100fe0 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9335899700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93301043f0 con 0x7f93301021e0 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9335899700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93301a1880 con 0x7f9330102b80 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93301021e0 0x7f93301035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93301021e0 0x7f93301035c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:53070/0 (socket says 172.21.15.164:53070) 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 -- 172.21.15.164:0/59833787 learned_addr learned my addr 172.21.15.164:0/59833787 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f932f7fe700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9330102b80 0x7f9330103b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f932effd700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9330100fe0 0x7f9330104f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:07.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 -- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9330102b80 msgr2=0x7f9330103b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9330102b80 0x7f9330103b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 -- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9330100fe0 msgr2=0x7f9330104f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9330100fe0 0x7f9330104f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 -- 172.21.15.164:0/59833787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93200095f0 con 0x7f93301021e0 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f932effd700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9330100fe0 0x7f9330104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9327fff700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93301021e0 0x7f93301035c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f931800b530 tx=0x7f931800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9334897700 1 -- 172.21.15.164:0/59833787 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318011830 con 0x7f93301021e0 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9334897700 1 -- 172.21.15.164:0/59833787 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f9318011e40 con 0x7f93301021e0 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93301a1b60 con 0x7f93301021e0 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9334897700 1 -- 172.21.15.164:0/59833787 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318010ea0 con 0x7f93301021e0 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.155+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93301a2060 con 0x7f93301021e0 2024-03-26T05:20:07.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.156+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9330068d40 con 0x7f93301021e0 2024-03-26T05:20:07.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.157+0000 7f9334897700 1 -- 172.21.15.164:0/59833787 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f9318011990 con 0x7f93301021e0 2024-03-26T05:20:07.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.157+0000 7f9334897700 1 --2- 172.21.15.164:0/59833787 >> v2:172.21.15.136:6800/2365933975 conn(0x7f931c06d990 0x7f931c06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:07.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.157+0000 7f9334897700 1 -- 172.21.15.164:0/59833787 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 3205+0+0 (secure 0 0 0) 0x7f931808c1d0 con 0x7f93301021e0 2024-03-26T05:20:07.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.158+0000 7f932effd700 1 --2- 172.21.15.164:0/59833787 >> v2:172.21.15.136:6800/2365933975 conn(0x7f931c06d990 0x7f931c06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:07.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.158+0000 7f932effd700 1 --2- 172.21.15.164:0/59833787 >> v2:172.21.15.136:6800/2365933975 conn(0x7f931c06d990 0x7f931c06fe50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f932000b580 tx=0x7f9320009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:20:07.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.162+0000 7f9334897700 1 -- 172.21.15.164:0/59833787 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f93180566b0 con 0x7f93301021e0 2024-03-26T05:20:07.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:07.364+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f93300fe110 con 0x7f931c06d990 2024-03-26T05:20:08.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:07 smithi164 ceph-mon[33554]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T05:20:08.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:07 smithi164 ceph-mon[33554]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:08.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:07 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:08.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:07 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:08.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:07 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:08.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[29560]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T05:20:08.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[29560]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:08.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:08.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:08.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:08.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[38106]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T05:20:08.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[38106]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:08.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:08.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:08.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:07 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:09.067 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:08 smithi164 ceph-mon[33554]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:09.224 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:08 smithi136 ceph-mon[29560]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:09.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:08 smithi136 ceph-mon[38106]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:10.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/2478900429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]: dispatch 2024-03-26T05:20:10.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]: dispatch 2024-03-26T05:20:10.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]': finished 2024-03-26T05:20:10.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:09 smithi164 ceph-mon[33554]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T05:20:10.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:09 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:10.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/2478900429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]: dispatch 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]: dispatch 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]': finished 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[29560]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/2478900429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]: dispatch 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]: dispatch 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dadc6cb-52c2-409a-b2ef-b6535d8ef50e"}]': finished 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[38106]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T05:20:10.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:09 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:11.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:10 smithi164 ceph-mon[33554]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T05:20:11.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/1534932915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:20:11.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:10 smithi136 ceph-mon[29560]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T05:20:11.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/1534932915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:20:11.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:10 smithi136 ceph-mon[38106]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T05:20:11.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/1534932915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:20:13.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:13 smithi164 ceph-mon[33554]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T05:20:14.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:13 smithi136 ceph-mon[38106]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T05:20:14.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:13 smithi136 ceph-mon[29560]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T05:20:15.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:15 smithi164 ceph-mon[33554]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T05:20:16.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:15 smithi136 ceph-mon[29560]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T05:20:16.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:15 smithi136 ceph-mon[38106]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T05:20:16.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T05:20:16.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:17.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T05:20:17.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:17.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T05:20:17.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:17.922 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:17 smithi164 ceph-mon[33554]: Deploying daemon osd.5 on smithi164 2024-03-26T05:20:17.922 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:17 smithi164 ceph-mon[33554]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-26T05:20:18.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:17 smithi136 ceph-mon[29560]: Deploying daemon osd.5 on smithi164 2024-03-26T05:20:18.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:17 smithi136 ceph-mon[29560]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-26T05:20:18.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:17 smithi136 ceph-mon[38106]: Deploying daemon osd.5 on smithi164 2024-03-26T05:20:18.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:17 smithi136 ceph-mon[38106]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-26T05:20:19.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:19 smithi164 ceph-mon[33554]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:20:19.935 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:19 smithi136 ceph-mon[29560]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:20:19.935 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:19 smithi136 ceph-mon[38106]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:20:21.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:21 smithi164 ceph-mon[33554]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:22.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:21 smithi136 ceph-mon[38106]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:22.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:21 smithi136 ceph-mon[29560]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:23.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:22 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:23.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:22 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:23.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:22 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:23.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:22 smithi136 ceph-mon[29560]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:23.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:22 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:23.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:22 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:23.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:22 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:23.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:22 smithi136 ceph-mon[38106]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:23.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:22 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:23.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:22 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:23.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:22 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:23.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:22 smithi164 ceph-mon[33554]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.176+0000 7f9334897700 1 -- 172.21.15.164:0/59833787 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f93300fe110 con 0x7f931c06d990 2024-03-26T05:20:25.166 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 5 on host 'smithi164' 2024-03-26T05:20:25.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.178+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 >> v2:172.21.15.136:6800/2365933975 conn(0x7f931c06d990 msgr2=0x7f931c06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.178+0000 7f9335899700 1 --2- 172.21.15.164:0/59833787 >> v2:172.21.15.136:6800/2365933975 conn(0x7f931c06d990 0x7f931c06fe50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f932000b580 tx=0x7f9320009f10 comp rx=0 tx=0).stop 2024-03-26T05:20:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93301021e0 msgr2=0x7f93301035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93301021e0 0x7f93301035c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f931800b530 tx=0x7f931800b8f0 comp rx=0 tx=0).stop 2024-03-26T05:20:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 shutdown_connections 2024-03-26T05:20:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 --2- 172.21.15.164:0/59833787 >> v2:172.21.15.136:6800/2365933975 conn(0x7f931c06d990 0x7f931c06fe50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9330102b80 0x7f9330103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9330100fe0 0x7f9330104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 --2- 172.21.15.164:0/59833787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93301021e0 0x7f93301035c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:25.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 >> 172.21.15.164:0/59833787 conn(0x7f93300fc5a0 msgr2=0x7f93300fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:20:25.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 shutdown_connections 2024-03-26T05:20:25.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:25.179+0000 7f9335899700 1 -- 172.21.15.164:0/59833787 wait complete. 2024-03-26T05:20:25.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:25 smithi164 ceph-mon[33554]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:25.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:25 smithi164 ceph-mon[33554]: from='osd.5 v2:172.21.15.164:6804/2714562868' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T05:20:25.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:25 smithi164 ceph-mon[33554]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T05:20:25.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:25 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:25.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:25 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:25.763 DEBUG:teuthology.orchestra.run.smithi164:osd.5> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T05:20:25.765 INFO:tasks.cephadm:Deploying osd.6 on smithi164 with /dev/vg_nvme/lv_2... 2024-03-26T05:20:25.765 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T05:20:26.050 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:26.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[29560]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:26.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[29560]: from='osd.5 v2:172.21.15.164:6804/2714562868' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T05:20:26.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[29560]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T05:20:26.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:26.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:26.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[38106]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:26.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[38106]: from='osd.5 v2:172.21.15.164:6804/2714562868' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T05:20:26.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[38106]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T05:20:26.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:26.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:25 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:26.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:26 smithi164 ceph-mon[33554]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T05:20:26.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:26 smithi164 ceph-mon[33554]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T05:20:26.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:26.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:26 smithi164 ceph-mon[33554]: from='osd.5 v2:172.21.15.164:6804/2714562868' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:26.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:26 smithi164 ceph-mon[33554]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:26.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:26.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:27.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[38106]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T05:20:27.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[38106]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T05:20:27.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:27.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[38106]: from='osd.5 v2:172.21.15.164:6804/2714562868' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:27.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[38106]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[29560]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[29560]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[29560]: from='osd.5 v2:172.21.15.164:6804/2714562868' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[29560]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:27.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:27.075 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:20:27.038+0000 7f29b7549700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T05:20:27.075 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:20:27.041+0000 7f29af334700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:20:27.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:27.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:27.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T05:20:27.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:27.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:27.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: from='osd.5 ' entity='osd.5' 2024-03-26T05:20:27.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: osd.5 v2:172.21.15.164:6804/2714562868 boot 2024-03-26T05:20:27.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T05:20:27.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:27 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:27.923 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T05:20:27.923 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T05:20:27.923 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-26T05:20:27.923 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-26T05:20:27.924 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129935 s, 807 MB/s 2024-03-26T05:20:27.924 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-26T05:20:28.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:28.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:28.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T05:20:28.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:28.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: from='osd.5 ' entity='osd.5' 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: osd.5 v2:172.21.15.164:6804/2714562868 boot 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: from='osd.5 ' entity='osd.5' 2024-03-26T05:20:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: osd.5 v2:172.21.15.164:6804/2714562868 boot 2024-03-26T05:20:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T05:20:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:27 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:20:28.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:28 smithi164 ceph-mon[33554]: purged_snaps scrub starts 2024-03-26T05:20:28.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:28 smithi164 ceph-mon[33554]: purged_snaps scrub ok 2024-03-26T05:20:29.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:28 smithi136 ceph-mon[29560]: purged_snaps scrub starts 2024-03-26T05:20:29.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:28 smithi136 ceph-mon[29560]: purged_snaps scrub ok 2024-03-26T05:20:29.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:28 smithi136 ceph-mon[38106]: purged_snaps scrub starts 2024-03-26T05:20:29.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:28 smithi136 ceph-mon[38106]: purged_snaps scrub ok 2024-03-26T05:20:29.574 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_2 2024-03-26T05:20:29.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:29 smithi164 ceph-mon[33554]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 493 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:29.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:29 smithi164 ceph-mon[33554]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T05:20:29.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:29 smithi164 ceph-mon[33554]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T05:20:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:29 smithi136 ceph-mon[29560]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 493 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:29 smithi136 ceph-mon[29560]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T05:20:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:29 smithi136 ceph-mon[29560]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T05:20:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:29 smithi136 ceph-mon[38106]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 493 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:29 smithi136 ceph-mon[38106]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T05:20:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:29 smithi136 ceph-mon[38106]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T05:20:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: Detected new or changed devices on smithi164 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi164 to 8544M 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:31.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:31.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:31.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:31.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 494 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:31.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:31 smithi164 ceph-mon[33554]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T05:20:31.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: Detected new or changed devices on smithi164 2024-03-26T05:20:31.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi164 to 8544M 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 494 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[29560]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T05:20:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: Detected new or changed devices on smithi164 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi164 to 8544M 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 494 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:31 smithi136 ceph-mon[38106]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T05:20:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.074+0000 7f60baddb700 1 -- 172.21.15.164:0/3174763200 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 msgr2=0x7f60b4104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.074+0000 7f60baddb700 1 --2- 172.21.15.164:0/3174763200 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 0x7f60b4104fd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009990 tx=0x7f60a8009ca0 comp rx=0 tx=0).stop 2024-03-26T05:20:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 -- 172.21.15.164:0/3174763200 shutdown_connections 2024-03-26T05:20:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 --2- 172.21.15.164:0/3174763200 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 0x7f60b4104fd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 --2- 172.21.15.164:0/3174763200 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60b4107e40 0x7f60b410a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 --2- 172.21.15.164:0/3174763200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b4105510 0x7f60b4107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 -- 172.21.15.164:0/3174763200 >> 172.21.15.164:0/3174763200 conn(0x7f60b40fc570 msgr2=0x7f60b40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 -- 172.21.15.164:0/3174763200 shutdown_connections 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 -- 172.21.15.164:0/3174763200 wait complete. 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 Processor -- start 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 -- start start 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 0x7f60b4197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b4105510 0x7f60b4197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60b4107e40 0x7f60b419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b419f2a0 con 0x7f60b4105510 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b419f3e0 con 0x7f60b4102be0 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60baddb700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b419f570 con 0x7f60b4107e40 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.075+0000 7f60b8b77700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 0x7f60b4197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b8b77700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 0x7f60b4197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54642/0 (socket says 172.21.15.164:54642) 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b8b77700 1 -- 172.21.15.164:0/2255846108 learned_addr learned my addr 172.21.15.164:0/2255846108 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:20:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b9378700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60b4107e40 0x7f60b419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b3fff700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b4105510 0x7f60b4197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b8b77700 1 -- 172.21.15.164:0/2255846108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60b4107e40 msgr2=0x7f60b419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b8b77700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60b4107e40 0x7f60b419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b8b77700 1 -- 172.21.15.164:0/2255846108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b4105510 msgr2=0x7f60b4197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b8b77700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b4105510 0x7f60b4197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b8b77700 1 -- 172.21.15.164:0/2255846108 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a80095f0 con 0x7f60b4102be0 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b8b77700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 0x7f60b4197710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f60a8006010 tx=0x7f60a8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b1ffb700 1 -- 172.21.15.164:0/2255846108 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a801d070 con 0x7f60b4102be0 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b419f7f0 con 0x7f60b4102be0 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b1ffb700 1 -- 172.21.15.164:0/2255846108 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f60a8004c90 con 0x7f60b4102be0 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60b1ffb700 1 -- 172.21.15.164:0/2255846108 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8003b20 con 0x7f60b4102be0 2024-03-26T05:20:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.076+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b419fc60 con 0x7f60b4102be0 2024-03-26T05:20:32.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.077+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b40fe0f0 con 0x7f60b4102be0 2024-03-26T05:20:32.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.078+0000 7f60b1ffb700 1 -- 172.21.15.164:0/2255846108 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f60a8005000 con 0x7f60b4102be0 2024-03-26T05:20:32.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.078+0000 7f60b1ffb700 1 --2- 172.21.15.164:0/2255846108 >> v2:172.21.15.136:6800/2365933975 conn(0x7f60a406d990 0x7f60a406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:32.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.079+0000 7f60b1ffb700 1 -- 172.21.15.164:0/2255846108 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 3497+0+0 (secure 0 0 0) 0x7f60a808d680 con 0x7f60b4102be0 2024-03-26T05:20:32.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.079+0000 7f60b3fff700 1 --2- 172.21.15.164:0/2255846108 >> v2:172.21.15.136:6800/2365933975 conn(0x7f60a406d990 0x7f60a406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:32.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.079+0000 7f60b3fff700 1 --2- 172.21.15.164:0/2255846108 >> v2:172.21.15.136:6800/2365933975 conn(0x7f60a406d990 0x7f60a406fe50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60a0005950 tx=0x7f60a000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:20:32.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.083+0000 7f60b1ffb700 1 -- 172.21.15.164:0/2255846108 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f60a8057ac0 con 0x7f60b4102be0 2024-03-26T05:20:32.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:32.289+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f60b40624c0 con 0x7f60a406d990 2024-03-26T05:20:32.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:32 smithi164 ceph-mon[33554]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T05:20:32.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:32 smithi164 ceph-mon[33554]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T05:20:32.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:32 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:32.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:32 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:32.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:32 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:32.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[29560]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T05:20:32.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[29560]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T05:20:32.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:32.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:32.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:32.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[38106]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T05:20:32.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[38106]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T05:20:32.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:32.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:32.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:32 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:33.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:33 smithi164 ceph-mon[33554]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:33.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:33 smithi164 ceph-mon[33554]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 494 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:33.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:33 smithi136 ceph-mon[29560]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:33.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:33 smithi136 ceph-mon[29560]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 494 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:33.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:33 smithi136 ceph-mon[38106]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:33.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:33 smithi136 ceph-mon[38106]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 494 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:34.631 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:34 smithi136 ceph-mon[29560]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T05:20:34.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:34 smithi164 ceph-mon[33554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T05:20:34.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:34 smithi164 ceph-mon[33554]: Cluster is now healthy 2024-03-26T05:20:35.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:34 smithi136 ceph-mon[29560]: Cluster is now healthy 2024-03-26T05:20:35.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:34 smithi136 ceph-mon[38106]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T05:20:35.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:34 smithi136 ceph-mon[38106]: Cluster is now healthy 2024-03-26T05:20:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:35 smithi164 ceph-mon[33554]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/2306856167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]: dispatch 2024-03-26T05:20:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]: dispatch 2024-03-26T05:20:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]': finished 2024-03-26T05:20:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:35 smithi164 ceph-mon[33554]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T05:20:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:35 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:35.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/1220999228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:20:36.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[38106]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:36.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/2306856167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]: dispatch 2024-03-26T05:20:36.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]: dispatch 2024-03-26T05:20:36.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]': finished 2024-03-26T05:20:36.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[38106]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/1220999228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[29560]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/2306856167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]: dispatch 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]: dispatch 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "018312be-ccea-4a85-9fa9-07154e0e0f15"}]': finished 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[29560]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:36.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/1220999228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:20:37.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:37 smithi164 ceph-mon[33554]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:20:38.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:37 smithi136 ceph-mon[29560]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:20:38.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:37 smithi136 ceph-mon[38106]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:20:39.966 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:39 smithi164 ceph-mon[33554]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T05:20:40.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:39 smithi136 ceph-mon[38106]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T05:20:40.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:39 smithi136 ceph-mon[29560]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T05:20:41.848 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:41 smithi164 ceph-mon[33554]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T05:20:41.848 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:41 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T05:20:41.848 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:41 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:42.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:41 smithi136 ceph-mon[38106]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T05:20:42.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T05:20:42.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:41 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:42.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:41 smithi136 ceph-mon[29560]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T05:20:42.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:41 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T05:20:42.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:41 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:42.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:42 smithi164 ceph-mon[33554]: Deploying daemon osd.6 on smithi164 2024-03-26T05:20:43.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:42 smithi136 ceph-mon[38106]: Deploying daemon osd.6 on smithi164 2024-03-26T05:20:43.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:42 smithi136 ceph-mon[29560]: Deploying daemon osd.6 on smithi164 2024-03-26T05:20:43.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:43 smithi164 ceph-mon[33554]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:20:44.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:43 smithi136 ceph-mon[38106]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:20:44.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:43 smithi136 ceph-mon[29560]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:20:45.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:45 smithi164 ceph-mon[33554]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:46.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:45 smithi136 ceph-mon[38106]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:46.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:45 smithi136 ceph-mon[29560]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:47.891 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:47 smithi164 ceph-mon[33554]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:47.891 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:47.891 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:47.891 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:48.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:47 smithi136 ceph-mon[38106]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:48.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:48.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:48.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:48.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:47 smithi136 ceph-mon[29560]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:48.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:48.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:48.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:49.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:49 smithi164 ceph-mon[33554]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:50.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:49 smithi136 ceph-mon[38106]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:50.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:49 smithi136 ceph-mon[29560]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:50.343 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 6 on host 'smithi164' 2024-03-26T05:20:50.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.352+0000 7f60b1ffb700 1 -- 172.21.15.164:0/2255846108 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f60b40624c0 con 0x7f60a406d990 2024-03-26T05:20:50.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.354+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 >> v2:172.21.15.136:6800/2365933975 conn(0x7f60a406d990 msgr2=0x7f60a406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:50.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.354+0000 7f60baddb700 1 --2- 172.21.15.164:0/2255846108 >> v2:172.21.15.136:6800/2365933975 conn(0x7f60a406d990 0x7f60a406fe50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60a0005950 tx=0x7f60a000a300 comp rx=0 tx=0).stop 2024-03-26T05:20:50.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.354+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 msgr2=0x7f60b4197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:50.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.354+0000 7f60baddb700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 0x7f60b4197710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f60a8006010 tx=0x7f60a8004730 comp rx=0 tx=0).stop 2024-03-26T05:20:50.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.355+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 shutdown_connections 2024-03-26T05:20:50.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.355+0000 7f60baddb700 1 --2- 172.21.15.164:0/2255846108 >> v2:172.21.15.136:6800/2365933975 conn(0x7f60a406d990 0x7f60a406fe50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:50.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.355+0000 7f60baddb700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60b4102be0 0x7f60b4197710 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:50.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.355+0000 7f60baddb700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60b4107e40 0x7f60b419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:50.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.355+0000 7f60baddb700 1 --2- 172.21.15.164:0/2255846108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b4105510 0x7f60b4197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:50.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.355+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 >> 172.21.15.164:0/2255846108 conn(0x7f60b40fc570 msgr2=0x7f60b4103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:20:50.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.355+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 shutdown_connections 2024-03-26T05:20:50.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:50.355+0000 7f60baddb700 1 -- 172.21.15.164:0/2255846108 wait complete. 2024-03-26T05:20:50.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:50 smithi164 ceph-mon[33554]: from='osd.6 v2:172.21.15.164:6808/268837773' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T05:20:50.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:50 smithi164 ceph-mon[33554]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T05:20:50.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:50 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:50.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:50 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:51.018 DEBUG:teuthology.orchestra.run.smithi164:osd.6> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T05:20:51.021 INFO:tasks.cephadm:Deploying osd.7 on smithi164 with /dev/vg_nvme/lv_1... 2024-03-26T05:20:51.021 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T05:20:51.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:50 smithi136 ceph-mon[29560]: from='osd.6 v2:172.21.15.164:6808/268837773' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T05:20:51.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:50 smithi136 ceph-mon[29560]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T05:20:51.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:50 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:51.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:50 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:51.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:50 smithi136 ceph-mon[38106]: from='osd.6 v2:172.21.15.164:6808/268837773' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T05:20:51.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:50 smithi136 ceph-mon[38106]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T05:20:51.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:50 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:51.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:50 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:51.305 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:51.678 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-mon[33554]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:51.678 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-mon[33554]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T05:20:51.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-mon[33554]: from='osd.6 v2:172.21.15.164:6808/268837773' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:51.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-mon[33554]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T05:20:51.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:51.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-mon[33554]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:51.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:51.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:51.967 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:20:51.687+0000 7f71b4d5f700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T05:20:51.967 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:20:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:20:51.695+0000 7f71b0b52700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:20:52.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[29560]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:52.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[29560]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T05:20:52.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[29560]: from='osd.6 v2:172.21.15.164:6808/268837773' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[29560]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[29560]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[38106]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[38106]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[38106]: from='osd.6 v2:172.21.15.164:6808/268837773' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[38106]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T05:20:52.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[38106]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:20:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:51 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:52.911 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:52 smithi164 ceph-mon[33554]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:52.911 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:52 smithi164 ceph-mon[33554]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T05:20:52.911 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:52 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:52.911 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:52 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:52.912 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:52 smithi164 ceph-mon[33554]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:53.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[29560]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:53.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[29560]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T05:20:53.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:53.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:53.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[29560]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:53.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[38106]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:20:53.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[38106]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T05:20:53.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:53.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:53.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:52 smithi136 ceph-mon[38106]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T05:20:53.377 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T05:20:53.377 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T05:20:53.377 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-26T05:20:53.377 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-26T05:20:53.377 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120552 s, 870 MB/s 2024-03-26T05:20:53.377 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-26T05:20:53.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:53 smithi164 ceph-mon[33554]: purged_snaps scrub starts 2024-03-26T05:20:53.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:53 smithi164 ceph-mon[33554]: purged_snaps scrub ok 2024-03-26T05:20:53.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:53 smithi164 ceph-mon[33554]: osd.6 v2:172.21.15.164:6808/268837773 boot 2024-03-26T05:20:53.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:53 smithi164 ceph-mon[33554]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T05:20:53.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:53 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:53.981 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_1 2024-03-26T05:20:54.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[29560]: purged_snaps scrub starts 2024-03-26T05:20:54.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[29560]: purged_snaps scrub ok 2024-03-26T05:20:54.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[29560]: osd.6 v2:172.21.15.164:6808/268837773 boot 2024-03-26T05:20:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[29560]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T05:20:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[38106]: purged_snaps scrub starts 2024-03-26T05:20:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[38106]: purged_snaps scrub ok 2024-03-26T05:20:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[38106]: osd.6 v2:172.21.15.164:6808/268837773 boot 2024-03-26T05:20:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[38106]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T05:20:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:53 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:20:54.405 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:20:54.942 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:54 smithi164 ceph-mon[33554]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T05:20:54.942 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:54 smithi164 ceph-mon[33554]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:54.942 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:54 smithi164 ceph-mon[33554]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T05:20:55.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:54 smithi136 ceph-mon[29560]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T05:20:55.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:54 smithi136 ceph-mon[29560]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:55.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:54 smithi136 ceph-mon[29560]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T05:20:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:54 smithi136 ceph-mon[38106]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T05:20:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:54 smithi136 ceph-mon[38106]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:55.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:54 smithi136 ceph-mon[38106]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T05:20:56.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 -- 172.21.15.164:0/142060871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7954103c70 msgr2=0x7f795410a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 --2- 172.21.15.164:0/142060871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7954103c70 0x7f795410a520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f794c00b0d0 tx=0x7f794c00b3e0 comp rx=0 tx=0).stop 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 -- 172.21.15.164:0/142060871 shutdown_connections 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 --2- 172.21.15.164:0/142060871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7954103c70 0x7f795410a520 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 --2- 172.21.15.164:0/142060871 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79541020d0 0x7f79541024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 --2- 172.21.15.164:0/142060871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79541032d0 0x7f7954103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 -- 172.21.15.164:0/142060871 >> 172.21.15.164:0/142060871 conn(0x7f79540fd690 msgr2=0x7f79540ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 -- 172.21.15.164:0/142060871 shutdown_connections 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.617+0000 7f795c618700 1 -- 172.21.15.164:0/142060871 wait complete. 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 Processor -- start 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 -- start start 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79541020d0 0x7f7954199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79541032d0 0x7f795419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:56.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7954103c70 0x7f79541a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79541a1660 con 0x7f7954103c70 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79541a17a0 con 0x7f79541032d0 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79541a1930 con 0x7f79541020d0 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79541032d0 0x7f795419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79541032d0 0x7f795419a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59640/0 (socket says 172.21.15.164:59640) 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 -- 172.21.15.164:0/3585503387 learned_addr learned my addr 172.21.15.164:0/3585503387 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795a3b4700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79541020d0 0x7f7954199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795abb5700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7954103c70 0x7f79541a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 -- 172.21.15.164:0/3585503387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79541020d0 msgr2=0x7f7954199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79541020d0 0x7f7954199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 -- 172.21.15.164:0/3585503387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7954103c70 msgr2=0x7f79541a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7954103c70 0x7f79541a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 -- 172.21.15.164:0/3585503387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c009d00 con 0x7f79541032d0 2024-03-26T05:20:56.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f7959bb3700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79541032d0 0x7f795419a010 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f794400c920 tx=0x7f794400cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:20:56.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f794b7fe700 1 -- 172.21.15.164:0/3585503387 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944004a30 con 0x7f79541032d0 2024-03-26T05:20:56.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f794b7fe700 1 -- 172.21.15.164:0/3585503387 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f79440054d0 con 0x7f79541032d0 2024-03-26T05:20:56.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79541a1c10 con 0x7f79541032d0 2024-03-26T05:20:56.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.618+0000 7f794b7fe700 1 -- 172.21.15.164:0/3585503387 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7944007bc0 con 0x7f79541032d0 2024-03-26T05:20:56.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.619+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79541a2160 con 0x7f79541032d0 2024-03-26T05:20:56.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.619+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7954068d40 con 0x7f79541032d0 2024-03-26T05:20:56.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.620+0000 7f794b7fe700 1 -- 172.21.15.164:0/3585503387 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f7944007d20 con 0x7f79541032d0 2024-03-26T05:20:56.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.620+0000 7f794b7fe700 1 --2- 172.21.15.164:0/3585503387 >> v2:172.21.15.136:6800/2365933975 conn(0x7f794006d990 0x7f794006fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:20:56.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.621+0000 7f794b7fe700 1 -- 172.21.15.164:0/3585503387 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 3789+0+0 (secure 0 0 0) 0x7f794408c3b0 con 0x7f79541032d0 2024-03-26T05:20:56.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.621+0000 7f795a3b4700 1 --2- 172.21.15.164:0/3585503387 >> v2:172.21.15.136:6800/2365933975 conn(0x7f794006d990 0x7f794006fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:20:56.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.621+0000 7f795a3b4700 1 --2- 172.21.15.164:0/3585503387 >> v2:172.21.15.136:6800/2365933975 conn(0x7f794006d990 0x7f794006fe50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7954103130 tx=0x7f7950009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:20:56.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.625+0000 7f794b7fe700 1 -- 172.21.15.164:0/3585503387 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f79440566d0 con 0x7f79541032d0 2024-03-26T05:20:56.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:20:56.840+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f79540752c0 con 0x7f794006d990 2024-03-26T05:20:56.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:56 smithi164 ceph-mon[33554]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T05:20:56.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:56 smithi164 ceph-mon[33554]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:57.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:56 smithi136 ceph-mon[29560]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T05:20:57.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:56 smithi136 ceph-mon[29560]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:57.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:56 smithi136 ceph-mon[38106]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T05:20:57.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:56 smithi136 ceph-mon[38106]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:57.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T05:20:57.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: Detected new or changed devices on smithi164 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi164 to 5696M 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:57.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: Detected new or changed devices on smithi164 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi164 to 5696M 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:20:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: Detected new or changed devices on smithi164 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi164 to 5696M 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:20:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:20:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:20:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:20:58.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:58 smithi164 ceph-mon[33554]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:59.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:58 smithi136 ceph-mon[29560]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:59.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:58 smithi136 ceph-mon[38106]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:20:59.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:59 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/186232950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]: dispatch 2024-03-26T05:20:59.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]: dispatch 2024-03-26T05:20:59.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]': finished 2024-03-26T05:20:59.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:59 smithi164 ceph-mon[33554]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T05:20:59.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:20:59 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:00.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/186232950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]: dispatch 2024-03-26T05:21:00.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]: dispatch 2024-03-26T05:21:00.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]': finished 2024-03-26T05:21:00.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[29560]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T05:21:00.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:00.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/186232950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]: dispatch 2024-03-26T05:21:00.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]: dispatch 2024-03-26T05:21:00.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd"}]': finished 2024-03-26T05:21:00.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[38106]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T05:21:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:20:59 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:00.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/2936224333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:21:00.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:00 smithi164 ceph-mon[33554]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-26T05:21:01.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/2936224333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:21:01.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:00 smithi136 ceph-mon[29560]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-26T05:21:01.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/2936224333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T05:21:01.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:00 smithi136 ceph-mon[38106]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-26T05:21:03.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:03 smithi164 ceph-mon[33554]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-26T05:21:04.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:03 smithi136 ceph-mon[29560]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-26T05:21:04.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:03 smithi136 ceph-mon[38106]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-26T05:21:05.924 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:05 smithi164 ceph-mon[33554]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T05:21:06.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:05 smithi136 ceph-mon[29560]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T05:21:06.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:05 smithi136 ceph-mon[38106]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T05:21:06.810 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T05:21:06.810 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:06 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:07.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T05:21:07.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:06 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:07.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T05:21:07.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:06 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:07.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:07 smithi164 ceph-mon[33554]: Deploying daemon osd.7 on smithi164 2024-03-26T05:21:07.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:07 smithi164 ceph-mon[33554]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-26T05:21:08.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:07 smithi136 ceph-mon[29560]: Deploying daemon osd.7 on smithi164 2024-03-26T05:21:08.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:07 smithi136 ceph-mon[29560]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-26T05:21:08.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:07 smithi136 ceph-mon[38106]: Deploying daemon osd.7 on smithi164 2024-03-26T05:21:08.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:07 smithi136 ceph-mon[38106]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-26T05:21:09.677 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:09 smithi164 ceph-mon[33554]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:21:09.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:09 smithi136 ceph-mon[38106]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:21:09.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:09 smithi136 ceph-mon[29560]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T05:21:11.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:11 smithi164 ceph-mon[33554]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:11.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:11 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:11.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:11 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:11.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:11 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:11 smithi136 ceph-mon[38106]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:11 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:11 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:11 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:12.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:11 smithi136 ceph-mon[29560]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:11 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:11 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:11 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:13.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:13 smithi164 ceph-mon[33554]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:14.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:13 smithi136 ceph-mon[29560]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:14.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:13 smithi136 ceph-mon[38106]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:14.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.852+0000 7f794b7fe700 1 -- 172.21.15.164:0/3585503387 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f79540752c0 con 0x7f794006d990 2024-03-26T05:21:14.842 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 7 on host 'smithi164' 2024-03-26T05:21:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 >> v2:172.21.15.136:6800/2365933975 conn(0x7f794006d990 msgr2=0x7f794006fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 --2- 172.21.15.164:0/3585503387 >> v2:172.21.15.136:6800/2365933975 conn(0x7f794006d990 0x7f794006fe50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7954103130 tx=0x7f7950009190 comp rx=0 tx=0).stop 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79541032d0 msgr2=0x7f795419a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79541032d0 0x7f795419a010 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f794400c920 tx=0x7f794400cc30 comp rx=0 tx=0).stop 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 shutdown_connections 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 --2- 172.21.15.164:0/3585503387 >> v2:172.21.15.136:6800/2365933975 conn(0x7f794006d990 0x7f794006fe50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7954103c70 0x7f79541a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79541020d0 0x7f7954199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 --2- 172.21.15.164:0/3585503387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79541032d0 0x7f795419a010 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 >> 172.21.15.164:0/3585503387 conn(0x7f79540fd690 msgr2=0x7f79540ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 shutdown_connections 2024-03-26T05:21:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:14.855+0000 7f795c618700 1 -- 172.21.15.164:0/3585503387 wait complete. 2024-03-26T05:21:15.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:15.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:15.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='osd.7 v2:172.21.15.164:6812/683372681' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:21:15.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:21:15.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:15.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:15 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:15.470 DEBUG:teuthology.orchestra.run.smithi164:osd.7> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T05:21:15.473 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T05:21:15.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T05:21:15.503 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.503 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='osd.7 v2:172.21.15.164:6812/683372681' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.504 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='osd.7 v2:172.21.15.164:6812/683372681' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:15.505 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:15 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:15.722 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:16.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:16 smithi164 ceph-mon[33554]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:21:16.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:16 smithi164 ceph-mon[33554]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T05:21:16.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:16 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:16.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:16 smithi164 ceph-mon[33554]: from='osd.7 v2:172.21.15.164:6812/683372681' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:21:16.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:16 smithi164 ceph-mon[33554]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:21:16.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[29560]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:21:16.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[29560]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T05:21:16.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:16.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[29560]: from='osd.7 v2:172.21.15.164:6812/683372681' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:21:16.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[29560]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:21:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[38106]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T05:21:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[38106]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T05:21:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[38106]: from='osd.7 v2:172.21.15.164:6812/683372681' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:21:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:16 smithi136 ceph-mon[38106]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T05:21:16.933 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:21:16 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:21:16.612+0000 7f2e87c89700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T05:21:16.933 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:21:16 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:21:16.618+0000 7f2e7f273700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T05:21:17.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:17 smithi164 ceph-mon[33554]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:21:17.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:17 smithi164 ceph-mon[33554]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T05:21:17.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:17.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:17 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:17.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:17 smithi164 ceph-mon[33554]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:17.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:17 smithi164 ceph-mon[33554]: from='osd.7 ' entity='osd.7' 2024-03-26T05:21:17.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 -- 172.21.15.136:0/963256957 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7381054d0 msgr2=0x7fa7381078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 --2- 172.21.15.136:0/963256957 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7381054d0 0x7fa7381078c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa7200098f0 tx=0x7fa720009c00 comp rx=0 tx=0).stop 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 -- 172.21.15.136:0/963256957 shutdown_connections 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 --2- 172.21.15.136:0/963256957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa738102ba0 0x7fa738104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 --2- 172.21.15.136:0/963256957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa738107e00 0x7fa73810a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 --2- 172.21.15.136:0/963256957 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7381054d0 0x7fa7381078c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 -- 172.21.15.136:0/963256957 >> 172.21.15.136:0/963256957 conn(0x7fa7380fc550 msgr2=0x7fa7380fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 -- 172.21.15.136:0/963256957 shutdown_connections 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 -- 172.21.15.136:0/963256957 wait complete. 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.349+0000 7fa73d890700 1 Processor -- start 2024-03-26T05:21:17.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa73d890700 1 -- start start 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa73d890700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa738102ba0 0x7fa738199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa73d890700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7381054d0 0x7fa738199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa73d890700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa738107e00 0x7fa7381a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa73d890700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7381a1640 con 0x7fa738102ba0 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa73d890700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7381a1780 con 0x7fa7381054d0 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa73d890700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7381a1910 con 0x7fa738107e00 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa738102ba0 0x7fa738199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa7377fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa738107e00 0x7fa7381a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa738102ba0 0x7fa738199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33348/0 (socket says 172.21.15.136:33348) 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 -- 172.21.15.136:0/1133276691 learned_addr learned my addr 172.21.15.136:0/1133276691 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa7367fc700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7381054d0 0x7fa738199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 -- 172.21.15.136:0/1133276691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa738107e00 msgr2=0x7fa7381a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa738107e00 0x7fa7381a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 -- 172.21.15.136:0/1133276691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7381054d0 msgr2=0x7fa738199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7381054d0 0x7fa738199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 -- 172.21.15.136:0/1133276691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280095f0 con 0x7fa738102ba0 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa7377fe700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa738107e00 0x7fa7381a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.350+0000 7fa736ffd700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa738102ba0 0x7fa738199ab0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa73000ba80 tx=0x7fa73000bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.351+0000 7fa73c88e700 1 -- 172.21.15.136:0/1133276691 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000c9b0 con 0x7fa738102ba0 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.351+0000 7fa73c88e700 1 -- 172.21.15.136:0/1133276691 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa730014440 con 0x7fa738102ba0 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.351+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa720009520 con 0x7fa738102ba0 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.351+0000 7fa73c88e700 1 -- 172.21.15.136:0/1133276691 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7300075d0 con 0x7fa738102ba0 2024-03-26T05:21:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.351+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7381a1fe0 con 0x7fa738102ba0 2024-03-26T05:21:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.352+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa738109980 con 0x7fa738102ba0 2024-03-26T05:21:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.352+0000 7fa73c88e700 1 -- 172.21.15.136:0/1133276691 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7fa73000cb10 con 0x7fa738102ba0 2024-03-26T05:21:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.352+0000 7fa73c88e700 1 --2- 172.21.15.136:0/1133276691 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa72406d990 0x7fa72406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.353+0000 7fa7367fc700 1 --2- 172.21.15.136:0/1133276691 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa72406d990 0x7fa72406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.353+0000 7fa73c88e700 1 -- 172.21.15.136:0/1133276691 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4114+0+0 (secure 0 0 0) 0x7fa730006e40 con 0x7fa738102ba0 2024-03-26T05:21:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.353+0000 7fa7367fc700 1 --2- 172.21.15.136:0/1133276691 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa72406d990 0x7fa72406fe50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa720009810 tx=0x7fa720019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:17.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.357+0000 7fa73c88e700 1 -- 172.21.15.136:0/1133276691 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa7300564c0 con 0x7fa738102ba0 2024-03-26T05:21:17.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.552+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa73810d240 con 0x7fa738102ba0 2024-03-26T05:21:17.542 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[38106]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:21:17.542 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[38106]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T05:21:17.542 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[38106]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[38106]: from='osd.7 ' entity='osd.7' 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[29560]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[29560]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[29560]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T05:21:17.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:17 smithi136 ceph-mon[29560]: from='osd.7 ' entity='osd.7' 2024-03-26T05:21:17.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.553+0000 7fa73c88e700 1 -- 172.21.15.136:0/1133276691 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v54) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa730059b80 con 0x7fa738102ba0 2024-03-26T05:21:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa72406d990 msgr2=0x7fa72406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 --2- 172.21.15.136:0/1133276691 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa72406d990 0x7fa72406fe50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa720009810 tx=0x7fa720019040 comp rx=0 tx=0).stop 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa738102ba0 msgr2=0x7fa738199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa738102ba0 0x7fa738199ab0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa73000ba80 tx=0x7fa73000bd90 comp rx=0 tx=0).stop 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 shutdown_connections 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 --2- 172.21.15.136:0/1133276691 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa72406d990 0x7fa72406fe50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa738107e00 0x7fa7381a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa738102ba0 0x7fa738199ab0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 --2- 172.21.15.136:0/1133276691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7381054d0 0x7fa738199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:17.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 >> 172.21.15.136:0/1133276691 conn(0x7fa7380fc550 msgr2=0x7fa738101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:17.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 shutdown_connections 2024-03-26T05:21:17.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:17.555+0000 7fa73d890700 1 -- 172.21.15.136:0/1133276691 wait complete. 2024-03-26T05:21:18.149 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1711430477,"num_in_osds":8,"osd_in_since":1711430459,"num_remapped_pgs":1} 2024-03-26T05:21:18.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T05:21:18.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:18 smithi164 ceph-mon[33554]: purged_snaps scrub starts 2024-03-26T05:21:18.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:18 smithi164 ceph-mon[33554]: purged_snaps scrub ok 2024-03-26T05:21:18.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:18 smithi164 ceph-mon[33554]: osd.7 v2:172.21.15.164:6812/683372681 boot 2024-03-26T05:21:18.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:18 smithi164 ceph-mon[33554]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T05:21:18.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:18 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:18.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1133276691' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T05:21:18.432 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[38106]: purged_snaps scrub starts 2024-03-26T05:21:18.432 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[38106]: purged_snaps scrub ok 2024-03-26T05:21:18.432 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[38106]: osd.7 v2:172.21.15.164:6812/683372681 boot 2024-03-26T05:21:18.432 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[38106]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T05:21:18.433 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:18.433 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1133276691' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T05:21:18.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[29560]: purged_snaps scrub starts 2024-03-26T05:21:18.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[29560]: purged_snaps scrub ok 2024-03-26T05:21:18.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[29560]: osd.7 v2:172.21.15.164:6812/683372681 boot 2024-03-26T05:21:18.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[29560]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T05:21:18.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:21:18.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1133276691' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T05:21:18.442 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:19.416 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: Detected new or changed devices on smithi164 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi164 to 4272M 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[29560]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: Detected new or changed devices on smithi164 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi164 to 4272M 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.418 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:19.419 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:19.419 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.419 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:19 smithi136 ceph-mon[38106]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:19.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: Detected new or changed devices on smithi164 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi164 to 4272M 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:19.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:19.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:19.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:19 smithi164 ceph-mon[33554]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:20.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.133+0000 7f6261a4c700 1 -- 172.21.15.136:0/315124946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c1070f0 msgr2=0x7f625c1074d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:20.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.133+0000 7f6261a4c700 1 --2- 172.21.15.136:0/315124946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c1070f0 0x7f625c1074d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f624c009990 tx=0x7f624c009ca0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.133+0000 7f6261a4c700 1 -- 172.21.15.136:0/315124946 shutdown_connections 2024-03-26T05:21:20.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.133+0000 7f6261a4c700 1 --2- 172.21.15.136:0/315124946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c101030 0x7f625c10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.133+0000 7f6261a4c700 1 --2- 172.21.15.136:0/315124946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c1070f0 0x7f625c1074d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.133+0000 7f6261a4c700 1 --2- 172.21.15.136:0/315124946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625c100670 0x7f625c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.133+0000 7f6261a4c700 1 -- 172.21.15.136:0/315124946 >> 172.21.15.136:0/315124946 conn(0x7f625c077580 msgr2=0x7f625c0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 -- 172.21.15.136:0/315124946 shutdown_connections 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 -- 172.21.15.136:0/315124946 wait complete. 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 Processor -- start 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 -- start start 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c100670 0x7f625c101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c101030 0x7f625c101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625c1070f0 0x7f625c1022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c1a61d0 con 0x7f625c1070f0 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c1a6340 con 0x7f625c101030 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.134+0000 7f6261a4c700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625c1a6500 con 0x7f625c100670 2024-03-26T05:21:20.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625c1070f0 0x7f625c1022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625affd700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c100670 0x7f625c101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625affd700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c100670 0x7f625c101850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:38276/0 (socket says 172.21.15.136:38276) 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625c1070f0 0x7f625c1022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33368/0 (socket says 172.21.15.136:33368) 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625affd700 1 -- 172.21.15.136:0/3957408223 learned_addr learned my addr 172.21.15.136:0/3957408223 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625a7fc700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c101030 0x7f625c101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625b7fe700 1 -- 172.21.15.136:0/3957408223 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c100670 msgr2=0x7f625c101850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625b7fe700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c100670 0x7f625c101850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625b7fe700 1 -- 172.21.15.136:0/3957408223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c101030 msgr2=0x7f625c101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625b7fe700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c101030 0x7f625c101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625b7fe700 1 -- 172.21.15.136:0/3957408223 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6250009520 con 0x7f625c1070f0 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625affd700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c100670 0x7f625c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f625b7fe700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625c1070f0 0x7f625c1022d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6254007ee0 tx=0x7f625400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f6260a4a700 1 -- 172.21.15.136:0/3957408223 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625400ddd0 con 0x7f625c1070f0 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624c0095f0 con 0x7f625c1070f0 2024-03-26T05:21:20.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f6260a4a700 1 -- 172.21.15.136:0/3957408223 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f625400f040 con 0x7f625c1070f0 2024-03-26T05:21:20.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625c1a6b50 con 0x7f625c1070f0 2024-03-26T05:21:20.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.135+0000 7f6260a4a700 1 -- 172.21.15.136:0/3957408223 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62540059b0 con 0x7f625c1070f0 2024-03-26T05:21:20.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.136+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625c068d40 con 0x7f625c1070f0 2024-03-26T05:21:20.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.137+0000 7f6260a4a700 1 -- 172.21.15.136:0/3957408223 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f6254005020 con 0x7f625c1070f0 2024-03-26T05:21:20.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.137+0000 7f6260a4a700 1 --2- 172.21.15.136:0/3957408223 >> v2:172.21.15.136:6800/2365933975 conn(0x7f624406d990 0x7f624406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:20.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.137+0000 7f6260a4a700 1 -- 172.21.15.136:0/3957408223 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4110+0+0 (secure 0 0 0) 0x7f625408c2a0 con 0x7f625c1070f0 2024-03-26T05:21:20.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.137+0000 7f625affd700 1 --2- 172.21.15.136:0/3957408223 >> v2:172.21.15.136:6800/2365933975 conn(0x7f624406d990 0x7f624406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:20.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.138+0000 7f625affd700 1 --2- 172.21.15.136:0/3957408223 >> v2:172.21.15.136:6800/2365933975 conn(0x7f624406d990 0x7f624406fe50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f624c00b2e0 tx=0x7f624c00b610 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:20.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.142+0000 7f6260a4a700 1 -- 172.21.15.136:0/3957408223 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6254056400 con 0x7f625c1070f0 2024-03-26T05:21:20.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.339+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f625c1118e0 con 0x7f625c1070f0 2024-03-26T05:21:20.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.339+0000 7f6260a4a700 1 -- 172.21.15.136:0/3957408223 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+11519 (secure 0 0 0) 0x7f6254059ac0 con 0x7f625c1070f0 2024-03-26T05:21:20.331 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:21:20.331 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":57,"fsid":"d69b39a8-eb2f-11ee-95d0-87774f69a715","created":"2024-03-26T05:15:55.712909+0000","modified":"2024-03-26T05:21:20.178400+0000","last_up_change":"2024-03-26T05:21:17.162399+0000","last_in_change":"2024-03-26T05:20:59.552150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T05:19:14.832191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4698473-d228-4853-b404-54b255f0a9d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":1923051194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1923051194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1923051194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":1923051194}]},"public_addr":"172.21.15.136:6801/1923051194","cluster_addr":"172.21.15.136:6802/1923051194","heartbeat_back_addr":"172.21.15.136:6804/1923051194","heartbeat_front_addr":"172.21.15.136:6803/1923051194","state":["exists","up"]},{"osd":1,"uuid":"800f61d3-4725-45b6-af8c-0653cb8b4c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":1223127568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1223127568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1223127568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":1223127568}]},"public_addr":"172.21.15.136:6805/1223127568","cluster_addr":"172.21.15.136:6806/1223127568","heartbeat_back_addr":"172.21.15.136:6808/1223127568","heartbeat_front_addr":"172.21.15.136:6807/1223127568","state":["exists","up"]},{"osd":2,"uuid":"0e60acb1-d576-40cf-9b67-28f11a3a2846","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":1411885928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1411885928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1411885928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":1411885928}]},"public_addr":"172.21.15.136:6809/1411885928","cluster_addr":"172.21.15.136:6810/1411885928","heartbeat_back_addr":"172.21.15.136:6812/1411885928","heartbeat_front_addr":"172.21.15.136:6811/1411885928","state":["exists","up"]},{"osd":3,"uuid":"c52d96a6-a66d-446b-aa35-891cc84fa99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":802386900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":802386900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":802386900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6815","nonce":802386900}]},"public_addr":"172.21.15.136:6813/802386900","cluster_addr":"172.21.15.136:6814/802386900","heartbeat_back_addr":"172.21.15.136:6816/802386900","heartbeat_front_addr":"172.21.15.136:6815/802386900","state":["exists","up"]},{"osd":4,"uuid":"a304c9c4-c2c8-4e3d-8292-48bc13c48ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":952209544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":952209544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":952209544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":952209544}]},"public_addr":"172.21.15.164:6800/952209544","cluster_addr":"172.21.15.164:6801/952209544","heartbeat_back_addr":"172.21.15.164:6803/952209544","heartbeat_front_addr":"172.21.15.164:6802/952209544","state":["exists","up"]},{"osd":5,"uuid":"0dadc6cb-52c2-409a-b2ef-b6535d8ef50e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2714562868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":2714562868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2714562868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2714562868}]},"public_addr":"172.21.15.164:6804/2714562868","cluster_addr":"172.21.15.164:6805/2714562868","heartbeat_back_addr":"172.21.15.164:6807/2714562868","heartbeat_front_addr":"172.21.15.164:6806/2714562868","state":["exists","up"]},{"osd":6,"uuid":"018312be-ccea-4a85-9fa9-07154e0e0f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":268837773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6809","nonce":268837773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":268837773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":268837773}]},"public_addr":"172.21.15.164:6808/268837773","cluster_addr":"172.21.15.164:6809/268837773","heartbeat_back_addr":"172.21.15.164:6811/268837773","heartbeat_front_addr":"172.21.15.164:6810/268837773","state":["exists","up"]},{"osd":7,"uuid":"f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":683372681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":683372681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":683372681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":683372681}]},"public_addr":"172.21.15.164:6812/683372681","cluster_addr":"172.21.15.164:6813/683372681","heartbeat_back_addr":"172.21.15.164:6815/683372681","heartbeat_front_addr":"172.21.15.164:6814/683372681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:18:21.637001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:18:47.061150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:19:12.511638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:19:38.406167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:01.579742+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:25.928726+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:51.081171+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:21:15.322145+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/3403923406":"2024-03-27T05:16:56.499053+0000","172.21.15.136:0/3432366522":"2024-03-27T05:16:56.499053+0000","172.21.15.136:0/3430411216":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/3098548195":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/404149158":"2024-03-27T05:16:29.672020+0000","172.21.15.136:6800/1232474076":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/747850292":"2024-03-27T05:16:56.499053+0000","172.21.15.136:6800/3861450553":"2024-03-27T05:16:56.499053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T05:21:20.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 >> v2:172.21.15.136:6800/2365933975 conn(0x7f624406d990 msgr2=0x7f624406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:20.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 --2- 172.21.15.136:0/3957408223 >> v2:172.21.15.136:6800/2365933975 conn(0x7f624406d990 0x7f624406fe50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f624c00b2e0 tx=0x7f624c00b610 comp rx=0 tx=0).stop 2024-03-26T05:21:20.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625c1070f0 msgr2=0x7f625c1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:20.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625c1070f0 0x7f625c1022d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6254007ee0 tx=0x7f625400d3b0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 shutdown_connections 2024-03-26T05:21:20.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 --2- 172.21.15.136:0/3957408223 >> v2:172.21.15.136:6800/2365933975 conn(0x7f624406d990 0x7f624406fe50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625c1070f0 0x7f625c1022d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f625c100670 0x7f625c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 --2- 172.21.15.136:0/3957408223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c101030 0x7f625c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:20.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 >> 172.21.15.136:0/3957408223 conn(0x7f625c077580 msgr2=0x7f625c104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:20.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 shutdown_connections 2024-03-26T05:21:20.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:20.342+0000 7f6261a4c700 1 -- 172.21.15.136:0/3957408223 wait complete. 2024-03-26T05:21:20.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:20 smithi164 ceph-mon[33554]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T05:21:20.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:20 smithi136 ceph-mon[38106]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T05:21:20.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:20 smithi136 ceph-mon[29560]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T05:21:21.034 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T05:19:14.832191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T05:21:21.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T05:21:21.275 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:21.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:21 smithi136 ceph-mon[29560]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T05:21:21.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3957408223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:21:21.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:21 smithi136 ceph-mon[29560]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T05:21:21.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:21 smithi136 ceph-mon[38106]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T05:21:21.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3957408223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:21:21.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:21 smithi136 ceph-mon[38106]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T05:21:21.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:21 smithi164 ceph-mon[33554]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T05:21:21.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3957408223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:21:21.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:21 smithi164 ceph-mon[33554]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T05:21:22.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:22 smithi164 ceph-mon[33554]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T05:21:22.485 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:22 smithi136 ceph-mon[38106]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T05:21:22.485 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:22 smithi136 ceph-mon[29560]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T05:21:23.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 -- 172.21.15.136:0/3007236848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 msgr2=0x7f4848100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:23.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 --2- 172.21.15.136:0/3007236848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 0x7f4848100af0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4838009990 tx=0x7f4838009ca0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 -- 172.21.15.136:0/3007236848 shutdown_connections 2024-03-26T05:21:23.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 --2- 172.21.15.136:0/3007236848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48481070f0 0x7f48481074d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 --2- 172.21.15.136:0/3007236848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4848101030 0x7f484810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 --2- 172.21.15.136:0/3007236848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 0x7f4848100af0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 -- 172.21.15.136:0/3007236848 >> 172.21.15.136:0/3007236848 conn(0x7f4848077580 msgr2=0x7f48480fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:23.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 -- 172.21.15.136:0/3007236848 shutdown_connections 2024-03-26T05:21:23.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.044+0000 7f485033a700 1 -- 172.21.15.136:0/3007236848 wait complete. 2024-03-26T05:21:23.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f485033a700 1 Processor -- start 2024-03-26T05:21:23.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f485033a700 1 -- start start 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f485033a700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 0x7f4848197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f485033a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4848101030 0x7f4848198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f485033a700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48481070f0 0x7f484819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f485033a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484819ccb0 con 0x7f4848101030 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f485033a700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484819ce20 con 0x7f48481070f0 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f485033a700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484819cfe0 con 0x7f4848100670 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f484e0d6700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 0x7f4848197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f484e0d6700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 0x7f4848197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:56530/0 (socket says 172.21.15.136:56530) 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f484e0d6700 1 -- 172.21.15.136:0/2444174882 learned_addr learned my addr 172.21.15.136:0/2444174882 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f484d8d5700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4848101030 0x7f4848198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.045+0000 7f484e8d7700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48481070f0 0x7f484819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f484e0d6700 1 -- 172.21.15.136:0/2444174882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48481070f0 msgr2=0x7f484819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f484e0d6700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48481070f0 0x7f484819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f484e0d6700 1 -- 172.21.15.136:0/2444174882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4848101030 msgr2=0x7f4848198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f484e0d6700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4848101030 0x7f4848198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f484e0d6700 1 -- 172.21.15.136:0/2444174882 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4840009c40 con 0x7f4848100670 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f484e8d7700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48481070f0 0x7f484819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f484e0d6700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 0x7f4848197e20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f484400c920 tx=0x7f484400cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f483f7fe700 1 -- 172.21.15.136:0/2444174882 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844007b00 con 0x7f4848100670 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f483f7fe700 1 -- 172.21.15.136:0/2444174882 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f484400ce70 con 0x7f4848100670 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48380095f0 con 0x7f4848100670 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f483f7fe700 1 -- 172.21.15.136:0/2444174882 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844005490 con 0x7f4848100670 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.046+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48481a4950 con 0x7f4848100670 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.047+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4848103990 con 0x7f4848100670 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.048+0000 7f483f7fe700 1 -- 172.21.15.136:0/2444174882 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f4844004750 con 0x7f4848100670 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.048+0000 7f483f7fe700 1 --2- 172.21.15.136:0/2444174882 >> v2:172.21.15.136:6800/2365933975 conn(0x7f483406d990 0x7f483406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.048+0000 7f484d8d5700 1 --2- 172.21.15.136:0/2444174882 >> v2:172.21.15.136:6800/2365933975 conn(0x7f483406d990 0x7f483406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:23.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.048+0000 7f483f7fe700 1 -- 172.21.15.136:0/2444174882 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7f484408c560 con 0x7f4848100670 2024-03-26T05:21:23.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.048+0000 7f484d8d5700 1 --2- 172.21.15.136:0/2444174882 >> v2:172.21.15.136:6800/2365933975 conn(0x7f483406d990 0x7f483406fe50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f48480ffbb0 tx=0x7f4838005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:23.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.053+0000 7f483f7fe700 1 -- 172.21.15.136:0/2444174882 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4844056750 con 0x7f4848100670 2024-03-26T05:21:23.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.246+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4848199160 con 0x7f4848100670 2024-03-26T05:21:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.247+0000 7f483f7fe700 1 -- 172.21.15.136:0/2444174882 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f484401d0a0 con 0x7f4848100670 2024-03-26T05:21:23.237 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-03-26T05:21:23.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 >> v2:172.21.15.136:6800/2365933975 conn(0x7f483406d990 msgr2=0x7f483406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 --2- 172.21.15.136:0/2444174882 >> v2:172.21.15.136:6800/2365933975 conn(0x7f483406d990 0x7f483406fe50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f48480ffbb0 tx=0x7f4838005fd0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 msgr2=0x7f4848197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 0x7f4848197e20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f484400c920 tx=0x7f484400cc30 comp rx=0 tx=0).stop 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 shutdown_connections 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 --2- 172.21.15.136:0/2444174882 >> v2:172.21.15.136:6800/2365933975 conn(0x7f483406d990 0x7f483406fe50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48481070f0 0x7f484819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4848100670 0x7f4848197e20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 --2- 172.21.15.136:0/2444174882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4848101030 0x7f4848198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 >> 172.21.15.136:0/2444174882 conn(0x7f4848077580 msgr2=0x7f4848104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 shutdown_connections 2024-03-26T05:21:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:23.249+0000 7f485033a700 1 -- 172.21.15.136:0/2444174882 wait complete. 2024-03-26T05:21:23.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:23 smithi164 ceph-mon[33554]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-26T05:21:23.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:23 smithi136 ceph-mon[29560]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-26T05:21:23.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:23 smithi136 ceph-mon[38106]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-26T05:21:23.907 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi136 2024-03-26T05:21:23.907 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi136=foo.a' 2024-03-26T05:21:24.160 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:24.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2444174882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T05:21:24.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2444174882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T05:21:24.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2444174882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T05:21:25.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:25 smithi164 ceph-mon[33554]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:21:25.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:25 smithi136 ceph-mon[29560]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:21:25.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:25 smithi136 ceph-mon[38106]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T05:21:25.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 -- 172.21.15.164:0/2430989441 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8100fb0 msgr2=0x7fc8a81013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 --2- 172.21.15.164:0/2430989441 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8100fb0 0x7fc8a81013d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc8a40099c0 tx=0x7fc8a4009cd0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 -- 172.21.15.164:0/2430989441 shutdown_connections 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 --2- 172.21.15.164:0/2430989441 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8100fb0 0x7fc8a81013d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 --2- 172.21.15.164:0/2430989441 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a8102b50 0x7fc8a8109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 --2- 172.21.15.164:0/2430989441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8a81021b0 0x7fc8a8102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 -- 172.21.15.164:0/2430989441 >> 172.21.15.164:0/2430989441 conn(0x7fc8a80fc570 msgr2=0x7fc8a80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 -- 172.21.15.164:0/2430989441 shutdown_connections 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 -- 172.21.15.164:0/2430989441 wait complete. 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 Processor -- start 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.700+0000 7fc8af741700 1 -- start start 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8af741700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8a8100fb0 0x7fc8a8197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8af741700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a81021b0 0x7fc8a8197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:25.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8af741700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a8102b50 0x7fc8a819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8af741700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a819f300 con 0x7fc8a8100fb0 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8af741700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a819f440 con 0x7fc8a81021b0 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8af741700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a819f5d0 con 0x7fc8a8102b50 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a81021b0 0x7fc8a8197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a81021b0 0x7fc8a8197cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33836/0 (socket says 172.21.15.164:33836) 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 -- 172.21.15.164:0/2713292968 learned_addr learned my addr 172.21.15.164:0/2713292968 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8adcde700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a8102b50 0x7fc8a819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8ad4dd700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8a8100fb0 0x7fc8a8197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 -- 172.21.15.164:0/2713292968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a8102b50 msgr2=0x7fc8a819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a8102b50 0x7fc8a819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 -- 172.21.15.164:0/2713292968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8a8100fb0 msgr2=0x7fc8a8197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8a8100fb0 0x7fc8a8197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 -- 172.21.15.164:0/2713292968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a40095f0 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8accdc700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a81021b0 0x7fc8a8197cb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc89800d950 tx=0x7fc89800dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc89e7fc700 1 -- 172.21.15.164:0/2713292968 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc898009b40 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a819f8b0 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc89e7fc700 1 -- 172.21.15.164:0/2713292968 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fc898009ca0 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc89e7fc700 1 -- 172.21.15.164:0/2713292968 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89800f620 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.701+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a819fdb0 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.702+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a8068d40 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.703+0000 7fc89e7fc700 1 -- 172.21.15.164:0/2713292968 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7fc898010460 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.703+0000 7fc89e7fc700 1 --2- 172.21.15.164:0/2713292968 >> v2:172.21.15.136:6800/2365933975 conn(0x7fc89406d990 0x7fc89406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.703+0000 7fc89e7fc700 1 -- 172.21.15.164:0/2713292968 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7fc89808c9f0 con 0x7fc8a81021b0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.703+0000 7fc8ad4dd700 1 --2- 172.21.15.164:0/2713292968 >> v2:172.21.15.136:6800/2365933975 conn(0x7fc89406d990 0x7fc89406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.704+0000 7fc8ad4dd700 1 --2- 172.21.15.164:0/2713292968 >> v2:172.21.15.136:6800/2365933975 conn(0x7fc89406d990 0x7fc89406fe50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400b580 tx=0x7fc8a4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.708+0000 7fc89e7fc700 1 -- 172.21.15.164:0/2713292968 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc898056b60 con 0x7fc8a81021b0 2024-03-26T05:21:25.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.915+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi136=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8a80fe130 con 0x7fc89406d990 2024-03-26T05:21:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.922+0000 7fc89e7fc700 1 -- 172.21.15.164:0/2713292968 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7fc8a80fe130 con 0x7fc89406d990 2024-03-26T05:21:25.912 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled rgw.foo.a update... 2024-03-26T05:21:25.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 >> v2:172.21.15.136:6800/2365933975 conn(0x7fc89406d990 msgr2=0x7fc89406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:25.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 --2- 172.21.15.164:0/2713292968 >> v2:172.21.15.136:6800/2365933975 conn(0x7fc89406d990 0x7fc89406fe50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400b580 tx=0x7fc8a4005fb0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a81021b0 msgr2=0x7fc8a8197cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:25.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a81021b0 0x7fc8a8197cb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc89800d950 tx=0x7fc89800dd10 comp rx=0 tx=0).stop 2024-03-26T05:21:25.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 shutdown_connections 2024-03-26T05:21:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 --2- 172.21.15.164:0/2713292968 >> v2:172.21.15.136:6800/2365933975 conn(0x7fc89406d990 0x7fc89406fe50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a8102b50 0x7fc8a819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8a8100fb0 0x7fc8a8197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 --2- 172.21.15.164:0/2713292968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a81021b0 0x7fc8a8197cb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 >> 172.21.15.164:0/2713292968 conn(0x7fc8a80fc570 msgr2=0x7fc8a80fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 shutdown_connections 2024-03-26T05:21:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:25.924+0000 7fc8af741700 1 -- 172.21.15.164:0/2713292968 wait complete. 2024-03-26T05:21:26.483 DEBUG:teuthology.orchestra.run.smithi136:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T05:21:26.485 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi164 2024-03-26T05:21:26.486 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T05:21:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:27.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi136=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:27.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T05:21:27.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:27.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:27.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:27.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T05:21:27.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T05:21:27.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:27.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: Deploying daemon rgw.foo.a on smithi136 2024-03-26T05:21:27.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:26 smithi164 ceph-mon[33554]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-26T05:21:27.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi136=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:27.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T05:21:27.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:27.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: Deploying daemon rgw.foo.a on smithi136 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[38106]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi136=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:27.188 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:27.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:27.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T05:21:27.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T05:21:27.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:27.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:27.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: Deploying daemon rgw.foo.a on smithi136 2024-03-26T05:21:27.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:26 smithi136 ceph-mon[29560]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-26T05:21:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 -- 172.21.15.164:0/4182092552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781032b0 msgr2=0x7f4f78103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/4182092552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781032b0 0x7f4f78103710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f680099c0 tx=0x7f4f68009cd0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 -- 172.21.15.164:0/4182092552 shutdown_connections 2024-03-26T05:21:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/4182092552 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f78103c50 0x7f4f7810a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/4182092552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f781020b0 0x7f4f781024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/4182092552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781032b0 0x7f4f78103710 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 -- 172.21.15.164:0/4182092552 >> 172.21.15.164:0/4182092552 conn(0x7f4f780fd690 msgr2=0x7f4f780ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 -- 172.21.15.164:0/4182092552 shutdown_connections 2024-03-26T05:21:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 -- 172.21.15.164:0/4182092552 wait complete. 2024-03-26T05:21:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.291+0000 7f4f7e83d700 1 Processor -- start 2024-03-26T05:21:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f7e83d700 1 -- start start 2024-03-26T05:21:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f7e83d700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781020b0 0x7f4f78197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f7e83d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f781032b0 0x7f4f78197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f7e83d700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f78103c50 0x7f4f7819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f7e83d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7819f2f0 con 0x7f4f781032b0 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f7e83d700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7819f430 con 0x7f4f781020b0 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f7e83d700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7819f5c0 con 0x7f4f78103c50 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781020b0 0x7f4f78197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781020b0 0x7f4f78197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33704/0 (socket says 172.21.15.164:33704) 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 -- 172.21.15.164:0/2724965097 learned_addr learned my addr 172.21.15.164:0/2724965097 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f777fe700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f781032b0 0x7f4f78197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f7cdda700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f78103c50 0x7f4f7819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 -- 172.21.15.164:0/2724965097 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f78103c50 msgr2=0x7f4f7819ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f78103c50 0x7f4f7819ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 -- 172.21.15.164:0/2724965097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f781032b0 msgr2=0x7f4f78197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f781032b0 0x7f4f78197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 -- 172.21.15.164:0/2724965097 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f680095f0 con 0x7f4f781020b0 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.292+0000 7f4f77fff700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781020b0 0x7f4f78197760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00d980 tx=0x7f4f6c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.293+0000 7f4f757fa700 1 -- 172.21.15.164:0/2724965097 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c004020 con 0x7f4f781020b0 2024-03-26T05:21:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.293+0000 7f4f757fa700 1 -- 172.21.15.164:0/2724965097 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f4f6c00f690 con 0x7f4f781020b0 2024-03-26T05:21:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.293+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7819f8a0 con 0x7f4f781020b0 2024-03-26T05:21:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.293+0000 7f4f757fa700 1 -- 172.21.15.164:0/2724965097 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c003a90 con 0x7f4f781020b0 2024-03-26T05:21:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.293+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7819fdf0 con 0x7f4f781020b0 2024-03-26T05:21:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.294+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f78109540 con 0x7f4f781020b0 2024-03-26T05:21:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.294+0000 7f4f757fa700 1 -- 172.21.15.164:0/2724965097 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f4f6c003bf0 con 0x7f4f781020b0 2024-03-26T05:21:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.294+0000 7f4f757fa700 1 --2- 172.21.15.164:0/2724965097 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4f6006d990 0x7f4f6006fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.295+0000 7f4f757fa700 1 -- 172.21.15.164:0/2724965097 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7f4f6c08c520 con 0x7f4f781020b0 2024-03-26T05:21:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.295+0000 7f4f777fe700 1 --2- 172.21.15.164:0/2724965097 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4f6006d990 0x7f4f6006fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.295+0000 7f4f777fe700 1 --2- 172.21.15.164:0/2724965097 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4f6006d990 0x7f4f6006fe50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009990 tx=0x7f4f68005ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:28.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.299+0000 7f4f757fa700 1 -- 172.21.15.164:0/2724965097 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4f6c056710 con 0x7f4f781020b0 2024-03-26T05:21:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.493+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f4f7810d380 con 0x7f4f781020b0 2024-03-26T05:21:28.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.799+0000 7f4f757fa700 1 -- 172.21.15.164:0/2724965097 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) v1 ==== 160+0+0 (secure 0 0 0) 0x7f4f6c059dd0 con 0x7f4f781020b0 2024-03-26T05:21:28.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.802+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4f6006d990 msgr2=0x7f4f6006fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:28.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.802+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/2724965097 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4f6006d990 0x7f4f6006fe50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009990 tx=0x7f4f68005ad0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.802+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781020b0 msgr2=0x7f4f78197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:28.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.802+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781020b0 0x7f4f78197760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00d980 tx=0x7f4f6c00dc90 comp rx=0 tx=0).stop 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.803+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 shutdown_connections 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.803+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/2724965097 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4f6006d990 0x7f4f6006fe50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.803+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f781020b0 0x7f4f78197760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.803+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f78103c50 0x7f4f7819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.803+0000 7f4f7e83d700 1 --2- 172.21.15.164:0/2724965097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f781032b0 0x7f4f78197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.803+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 >> 172.21.15.164:0/2724965097 conn(0x7f4f780fd690 msgr2=0x7f4f78106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.803+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 shutdown_connections 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:28.803+0000 7f4f7e83d700 1 -- 172.21.15.164:0/2724965097 wait complete. 2024-03-26T05:21:28.793 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'datapool' created 2024-03-26T05:21:29.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/2724965097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:21:29.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:21:29.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:28 smithi136 ceph-mon[29560]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:29.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/2724965097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:21:29.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:21:29.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:28 smithi136 ceph-mon[38106]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:29.074 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:21:29 smithi136 podman[64240]: 2024-03-26 05:21:29.005940139 +0000 UTC m=+0.138647683 container create 522855bae350c0fb261f7994df108b50629cd819cbd16a7e2d3becdc92da64d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=) 2024-03-26T05:21:29.111 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/2724965097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:21:29.111 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T05:21:29.111 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:28 smithi164 ceph-mon[33554]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:29.425 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T05:21:29.466 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:21:29 smithi136 podman[64240]: 2024-03-26 05:21:29.224899567 +0000 UTC m=+0.357607120 container init 522855bae350c0fb261f7994df108b50629cd819cbd16a7e2d3becdc92da64d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T05:21:29.466 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:21:29 smithi136 podman[64240]: 2024-03-26 05:21:29.22766305 +0000 UTC m=+0.360370604 container start 522855bae350c0fb261f7994df108b50629cd819cbd16a7e2d3becdc92da64d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T05:21:29.466 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:21:29 smithi136 bash[64240]: 522855bae350c0fb261f7994df108b50629cd819cbd16a7e2d3becdc92da64d7 2024-03-26T05:21:29.466 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:21:29 smithi136 systemd[1]: Started Ceph rgw.foo.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:21:29.674 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T05:21:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T05:21:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T05:21:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:29 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:30.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T05:21:30.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T05:21:30.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: Saving service rgw.foo.a spec with placement smithi136=foo.a;count:1 2024-03-26T05:21:30.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:30.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:29 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:31.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:30 smithi164 ceph-mon[33554]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T05:21:31.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2736442546' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T05:21:31.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:30 smithi164 ceph-mon[33554]: pgmap v187: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:31.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:30 smithi136 ceph-mon[38106]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T05:21:31.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2736442546' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T05:21:31.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:30 smithi136 ceph-mon[38106]: pgmap v187: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:31.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:30 smithi136 ceph-mon[29560]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T05:21:31.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2736442546' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T05:21:31.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:30 smithi136 ceph-mon[29560]: pgmap v187: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:21:32.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:21:32.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2736442546' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T05:21:32.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T05:21:32.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:32.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:32.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: Checking dashboard <-> RGW credentials 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/312311625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2736442546' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:32.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: Checking dashboard <-> RGW credentials 2024-03-26T05:21:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/312311625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:21:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:21:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:31 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:32.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:21:32.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2736442546' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T05:21:32.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T05:21:32.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:32.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:32.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:32.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:32.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: Checking dashboard <-> RGW credentials 2024-03-26T05:21:32.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/312311625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:21:32.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T05:21:32.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:31 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:33.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T05:21:33.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[38106]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T05:21:33.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[38106]: pgmap v190: 68 pgs: 6 creating+peering, 9 active+clean, 53 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:21:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T05:21:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[29560]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T05:21:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:32 smithi136 ceph-mon[29560]: pgmap v190: 68 pgs: 6 creating+peering, 9 active+clean, 53 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:21:33.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T05:21:33.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:32 smithi164 ceph-mon[33554]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T05:21:33.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T05:21:33.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:32 smithi164 ceph-mon[33554]: pgmap v190: 68 pgs: 6 creating+peering, 9 active+clean, 53 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:21:34.137 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:21:34.137 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:21:34.137 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:33 smithi164 ceph-mon[33554]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T05:21:34.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:21:34.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:21:34.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:33 smithi136 ceph-mon[29560]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T05:21:34.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:21:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T05:21:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:33 smithi136 ceph-mon[38106]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T05:21:34.457 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.164 --placement '1;smithi164=iscsi.a' 2024-03-26T05:21:34.695 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:35.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:34 smithi164 ceph-mon[33554]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T05:21:35.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:34 smithi164 ceph-mon[33554]: pgmap v193: 100 pgs: 15 creating+peering, 44 active+clean, 41 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:21:35.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:21:35.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:21:35.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:34 smithi164 ceph-mon[33554]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[29560]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[29560]: pgmap v193: 100 pgs: 15 creating+peering, 44 active+clean, 41 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[29560]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[38106]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T05:21:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T05:21:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[38106]: pgmap v193: 100 pgs: 15 creating+peering, 44 active+clean, 41 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:21:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:21:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T05:21:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:34 smithi136 ceph-mon[38106]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 -- 172.21.15.164:0/144828474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981032d0 msgr2=0x7f7d98103730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 --2- 172.21.15.164:0/144828474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981032d0 0x7f7d98103730 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d880099c0 tx=0x7f7d88009cd0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 -- 172.21.15.164:0/144828474 shutdown_connections 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 --2- 172.21.15.164:0/144828474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d98103c70 0x7f7d9810a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 --2- 172.21.15.164:0/144828474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d981020d0 0x7f7d981024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 --2- 172.21.15.164:0/144828474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981032d0 0x7f7d98103730 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 -- 172.21.15.164:0/144828474 >> 172.21.15.164:0/144828474 conn(0x7f7d980fd690 msgr2=0x7f7d980ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 -- 172.21.15.164:0/144828474 shutdown_connections 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.202+0000 7f7d9f546700 1 -- 172.21.15.164:0/144828474 wait complete. 2024-03-26T05:21:36.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9f546700 1 Processor -- start 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9f546700 1 -- start start 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9f546700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981020d0 0x7f7d981999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9f546700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d981032d0 0x7f7d98199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9f546700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d98103c70 0x7f7d981a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9f546700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d981a1500 con 0x7f7d98103c70 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9f546700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d981a1640 con 0x7f7d981020d0 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9f546700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d981a17d0 con 0x7f7d981032d0 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9d2e2700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981020d0 0x7f7d981999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9d2e2700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981020d0 0x7f7d981999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33732/0 (socket says 172.21.15.164:33732) 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9d2e2700 1 -- 172.21.15.164:0/2156459423 learned_addr learned my addr 172.21.15.164:0/2156459423 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9cae1700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d981032d0 0x7f7d98199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9dae3700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d98103c70 0x7f7d981a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9d2e2700 1 -- 172.21.15.164:0/2156459423 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d981032d0 msgr2=0x7f7d98199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9d2e2700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d981032d0 0x7f7d98199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9d2e2700 1 -- 172.21.15.164:0/2156459423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d98103c70 msgr2=0x7f7d981a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:36.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9d2e2700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d98103c70 0x7f7d981a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.203+0000 7f7d9d2e2700 1 -- 172.21.15.164:0/2156459423 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880095f0 con 0x7f7d981020d0 2024-03-26T05:21:36.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.204+0000 7f7d9d2e2700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981020d0 0x7f7d981999c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400b500 tx=0x7f7d9400b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:36.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.204+0000 7f7d8e7fc700 1 -- 172.21.15.164:0/2156459423 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d94011830 con 0x7f7d981020d0 2024-03-26T05:21:36.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.204+0000 7f7d8e7fc700 1 -- 172.21.15.164:0/2156459423 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f7d94011e40 con 0x7f7d981020d0 2024-03-26T05:21:36.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.204+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d981a1ab0 con 0x7f7d981020d0 2024-03-26T05:21:36.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.204+0000 7f7d8e7fc700 1 -- 172.21.15.164:0/2156459423 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9400f2e0 con 0x7f7d981020d0 2024-03-26T05:21:36.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.204+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d981a1fb0 con 0x7f7d981020d0 2024-03-26T05:21:36.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.205+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d98050590 con 0x7f7d981020d0 2024-03-26T05:21:36.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.205+0000 7f7d8e7fc700 1 -- 172.21.15.164:0/2156459423 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f7d94011990 con 0x7f7d981020d0 2024-03-26T05:21:36.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.206+0000 7f7d8e7fc700 1 --2- 172.21.15.164:0/2156459423 >> v2:172.21.15.136:6800/2365933975 conn(0x7f7d8406da30 0x7f7d8406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:36.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.206+0000 7f7d8e7fc700 1 -- 172.21.15.164:0/2156459423 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f7d9408cc60 con 0x7f7d981020d0 2024-03-26T05:21:36.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.206+0000 7f7d9cae1700 1 --2- 172.21.15.164:0/2156459423 >> v2:172.21.15.136:6800/2365933975 conn(0x7f7d8406da30 0x7f7d8406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:36.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.206+0000 7f7d9cae1700 1 --2- 172.21.15.164:0/2156459423 >> v2:172.21.15.136:6800/2365933975 conn(0x7f7d8406da30 0x7f7d8406fef0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7d88000c00 tx=0x7f7d88005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:36.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.210+0000 7f7d8e7fc700 1 -- 172.21.15.164:0/2156459423 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7d940566b0 con 0x7f7d981020d0 2024-03-26T05:21:36.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.417+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.164", "placement": "1;smithi164=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d980752c0 con 0x7f7d8406da30 2024-03-26T05:21:36.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.425+0000 7f7d8e7fc700 1 -- 172.21.15.164:0/2156459423 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7f7d980752c0 con 0x7f7d8406da30 2024-03-26T05:21:36.415 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled iscsi.datapool update... 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 >> v2:172.21.15.136:6800/2365933975 conn(0x7f7d8406da30 msgr2=0x7f7d8406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 --2- 172.21.15.164:0/2156459423 >> v2:172.21.15.136:6800/2365933975 conn(0x7f7d8406da30 0x7f7d8406fef0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7d88000c00 tx=0x7f7d88005fb0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981020d0 msgr2=0x7f7d981999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981020d0 0x7f7d981999c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400b500 tx=0x7f7d9400b8c0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 shutdown_connections 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 --2- 172.21.15.164:0/2156459423 >> v2:172.21.15.136:6800/2365933975 conn(0x7f7d8406da30 0x7f7d8406fef0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d981020d0 0x7f7d981999c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d98103c70 0x7f7d981a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 --2- 172.21.15.164:0/2156459423 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d981032d0 0x7f7d98199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:36.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 >> 172.21.15.164:0/2156459423 conn(0x7f7d980fd690 msgr2=0x7f7d980ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:36.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 shutdown_connections 2024-03-26T05:21:36.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:36.427+0000 7f7d9f546700 1 -- 172.21.15.164:0/2156459423 wait complete. 2024-03-26T05:21:37.039 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T05:21:37.039 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:21:37.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T05:21:37.075 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:21:37.075 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T05:21:37.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:36 smithi164 ceph-mon[33554]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T05:21:37.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:36 smithi164 ceph-mon[33554]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.164", "placement": "1;smithi164=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:37.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:36 smithi164 ceph-mon[33554]: Saving service iscsi.datapool spec with placement smithi164=iscsi.a;count:1 2024-03-26T05:21:37.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:36 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:37.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:36 smithi164 ceph-mon[33554]: pgmap v196: 132 pgs: 13 creating+peering, 75 active+clean, 44 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:21:37.104 DEBUG:teuthology.orchestra.run.smithi164:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T05:21:37.147 INFO:tasks.cephadm:Adding prometheus.a on smithi164 2024-03-26T05:21:37.147 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi164=a' 2024-03-26T05:21:37.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[38106]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T05:21:37.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[38106]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.164", "placement": "1;smithi164=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[38106]: Saving service iscsi.datapool spec with placement smithi164=iscsi.a;count:1 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[38106]: pgmap v196: 132 pgs: 13 creating+peering, 75 active+clean, 44 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[29560]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[29560]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.164", "placement": "1;smithi164=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[29560]: Saving service iscsi.datapool spec with placement smithi164=iscsi.a;count:1 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:36 smithi136 ceph-mon[29560]: pgmap v196: 132 pgs: 13 creating+peering, 75 active+clean, 44 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:21:37.423 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:37.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:21:37.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:21:37.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T05:21:37.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:21:38.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:21:38.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:37 smithi164 ceph-mon[33554]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T05:21:38.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:21:38.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:21:38.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[29560]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2659226996' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T05:21:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1134964439' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:21:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T05:21:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:37 smithi136 ceph-mon[38106]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T05:21:39.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 -- 172.21.15.164:0/2832905754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08701032b0 msgr2=0x7f0870103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:39.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 --2- 172.21.15.164:0/2832905754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08701032b0 0x7f0870103710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08600099c0 tx=0x7f0860009cd0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 -- 172.21.15.164:0/2832905754 shutdown_connections 2024-03-26T05:21:39.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 --2- 172.21.15.164:0/2832905754 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0870103c50 0x7f087010a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 --2- 172.21.15.164:0/2832905754 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08701020b0 0x7f08701024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 --2- 172.21.15.164:0/2832905754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08701032b0 0x7f0870103710 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 -- 172.21.15.164:0/2832905754 >> 172.21.15.164:0/2832905754 conn(0x7f08700fd690 msgr2=0x7f08700ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:39.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 -- 172.21.15.164:0/2832905754 shutdown_connections 2024-03-26T05:21:39.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.044+0000 7f08785c9700 1 -- 172.21.15.164:0/2832905754 wait complete. 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f08785c9700 1 Processor -- start 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f08785c9700 1 -- start start 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f08785c9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08701020b0 0x7f0870197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f08785c9700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08701032b0 0x7f0870197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f08785c9700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0870103c50 0x7f087019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f08785c9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087019f2e0 con 0x7f08701020b0 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f08785c9700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087019f420 con 0x7f0870103c50 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f08785c9700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087019f5b0 con 0x7f08701032b0 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f0876b66700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0870103c50 0x7f087019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f0876b66700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0870103c50 0x7f087019ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:50142/0 (socket says 172.21.15.164:50142) 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f0876b66700 1 -- 172.21.15.164:0/3576105115 learned_addr learned my addr 172.21.15.164:0/3576105115 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f0876365700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08701020b0 0x7f0870197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:39.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.045+0000 7f0875b64700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08701032b0 0x7f0870197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f0876b66700 1 -- 172.21.15.164:0/3576105115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08701032b0 msgr2=0x7f0870197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f0876b66700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08701032b0 0x7f0870197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f0876b66700 1 -- 172.21.15.164:0/3576105115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08701020b0 msgr2=0x7f0870197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f0876b66700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08701020b0 0x7f0870197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f0876b66700 1 -- 172.21.15.164:0/3576105115 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600095f0 con 0x7f0870103c50 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f0876b66700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0870103c50 0x7f087019ed10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0868007ee0 tx=0x7f086800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f08677fe700 1 -- 172.21.15.164:0/3576105115 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086800dd20 con 0x7f0870103c50 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087019f890 con 0x7f0870103c50 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f08677fe700 1 -- 172.21.15.164:0/3576105115 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f086800f040 con 0x7f0870103c50 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f08677fe700 1 -- 172.21.15.164:0/3576105115 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0868004840 con 0x7f0870103c50 2024-03-26T05:21:39.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.046+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087019fd90 con 0x7f0870103c50 2024-03-26T05:21:39.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.047+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0870109560 con 0x7f0870103c50 2024-03-26T05:21:39.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.047+0000 7f08677fe700 1 -- 172.21.15.164:0/3576105115 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 91364+0+0 (secure 0 0 0) 0x7f0868003a00 con 0x7f0870103c50 2024-03-26T05:21:39.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.048+0000 7f08677fe700 1 --2- 172.21.15.164:0/3576105115 >> v2:172.21.15.136:6800/2365933975 conn(0x7f085c06d990 0x7f085c06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:39.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.048+0000 7f08677fe700 1 -- 172.21.15.164:0/3576105115 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f086808c940 con 0x7f0870103c50 2024-03-26T05:21:39.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.048+0000 7f0876365700 1 --2- 172.21.15.164:0/3576105115 >> v2:172.21.15.136:6800/2365933975 conn(0x7f085c06d990 0x7f085c06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:39.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.048+0000 7f0876365700 1 --2- 172.21.15.164:0/3576105115 >> v2:172.21.15.136:6800/2365933975 conn(0x7f085c06d990 0x7f085c06fe50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0870103110 tx=0x7f086c007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:39.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.052+0000 7f08677fe700 1 -- 172.21.15.164:0/3576105115 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0868056370 con 0x7f0870103c50 2024-03-26T05:21:39.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.266+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f087010d400 con 0x7f085c06d990 2024-03-26T05:21:39.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.274+0000 7f08677fe700 1 -- 172.21.15.164:0/3576105115 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f087010d400 con 0x7f085c06d990 2024-03-26T05:21:39.264 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled prometheus update... 2024-03-26T05:21:39.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 >> v2:172.21.15.136:6800/2365933975 conn(0x7f085c06d990 msgr2=0x7f085c06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:39.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 --2- 172.21.15.164:0/3576105115 >> v2:172.21.15.136:6800/2365933975 conn(0x7f085c06d990 0x7f085c06fe50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0870103110 tx=0x7f086c007400 comp rx=0 tx=0).stop 2024-03-26T05:21:39.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0870103c50 msgr2=0x7f087019ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:39.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0870103c50 0x7f087019ed10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0868007ee0 tx=0x7f086800d3b0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 shutdown_connections 2024-03-26T05:21:39.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 --2- 172.21.15.164:0/3576105115 >> v2:172.21.15.136:6800/2365933975 conn(0x7f085c06d990 0x7f085c06fe50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0870103c50 0x7f087019ed10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08701020b0 0x7f0870197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 --2- 172.21.15.164:0/3576105115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08701032b0 0x7f0870197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:39.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 >> 172.21.15.164:0/3576105115 conn(0x7f08700fd690 msgr2=0x7f08701066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:39.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 shutdown_connections 2024-03-26T05:21:39.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:39.277+0000 7f08785c9700 1 -- 172.21.15.164:0/3576105115 wait complete. 2024-03-26T05:21:39.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: Deploying daemon iscsi.iscsi.a on smithi164 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: pgmap v199: 132 pgs: 11 creating+peering, 96 active+clean, 25 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[29560]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T05:21:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T05:21:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: Deploying daemon iscsi.iscsi.a on smithi164 2024-03-26T05:21:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: pgmap v199: 132 pgs: 11 creating+peering, 96 active+clean, 25 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:21:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:39 smithi136 ceph-mon[38106]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:21:39.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:39.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T05:21:39.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T05:21:39.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:39.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: Deploying daemon iscsi.iscsi.a on smithi164 2024-03-26T05:21:39.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: pgmap v199: 132 pgs: 11 creating+peering, 96 active+clean, 25 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:21:39.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:39 smithi164 ceph-mon[33554]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:21:40.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[29560]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:21:40.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[29560]: Cluster is now healthy 2024-03-26T05:21:40.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[29560]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:40.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[29560]: Saving service prometheus spec with placement smithi164=a;count:1 2024-03-26T05:21:40.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:40.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[38106]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:21:40.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[38106]: Cluster is now healthy 2024-03-26T05:21:40.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[38106]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:40.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[38106]: Saving service prometheus spec with placement smithi164=a;count:1 2024-03-26T05:21:40.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:40 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:40.366 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:40 smithi164 ceph-mon[33554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T05:21:40.366 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:40 smithi164 ceph-mon[33554]: Cluster is now healthy 2024-03-26T05:21:40.366 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:40 smithi164 ceph-mon[33554]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:40.366 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:40 smithi164 ceph-mon[33554]: Saving service prometheus spec with placement smithi164=a;count:1 2024-03-26T05:21:40.366 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:40 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:40.915 DEBUG:teuthology.orchestra.run.smithi164:prometheus.a> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T05:21:40.917 INFO:tasks.cephadm:Adding node-exporter.a on smithi136 2024-03-26T05:21:40.917 INFO:tasks.cephadm:Adding node-exporter.b on smithi164 2024-03-26T05:21:40.918 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi136=a;smithi164=b' 2024-03-26T05:21:41.313 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:41 smithi164 ceph-mon[33554]: pgmap v200: 132 pgs: 125 active+clean, 7 unknown; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.9 KiB/s wr, 193 op/s 2024-03-26T05:21:41.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 podman[60076]: 2024-03-26 05:21:41.125154499 +0000 UTC m=+0.017707086 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:21:41.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 podman[60076]: 2024-03-26 05:21:41.324602492 +0000 UTC m=+0.217155067 container create 0b4fe70eb4bce817896a66f0715e0a29bb26681baf42c439e38fbab36f8a3222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124) 2024-03-26T05:21:41.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:41 smithi136 ceph-mon[38106]: pgmap v200: 132 pgs: 125 active+clean, 7 unknown; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.9 KiB/s wr, 193 op/s 2024-03-26T05:21:41.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:41 smithi136 ceph-mon[29560]: pgmap v200: 132 pgs: 125 active+clean, 7 unknown; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.9 KiB/s wr, 193 op/s 2024-03-26T05:21:41.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 podman[60076]: 2024-03-26 05:21:41.543579356 +0000 UTC m=+0.436131931 container init 0b4fe70eb4bce817896a66f0715e0a29bb26681baf42c439e38fbab36f8a3222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T05:21:41.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 podman[60076]: 2024-03-26 05:21:41.546543711 +0000 UTC m=+0.439096287 container start 0b4fe70eb4bce817896a66f0715e0a29bb26681baf42c439e38fbab36f8a3222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T05:21:41.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 bash[60076]: 0b4fe70eb4bce817896a66f0715e0a29bb26681baf42c439e38fbab36f8a3222 2024-03-26T05:21:41.959 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 systemd[1]: Started Ceph iscsi.iscsi.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:21:41.959 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug Started the configuration object watcher 2024-03-26T05:21:41.959 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug Checking for config object changes every 1s 2024-03-26T05:21:41.959 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug Processing osd blocklist entries for this node 2024-03-26T05:21:42.193 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:42.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug Reading the configuration object to update local LIO configuration 2024-03-26T05:21:42.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug Configuration does not have an entry for this host(smithi164.front.sepia.ceph.com) - nothing to define to LIO 2024-03-26T05:21:42.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-26T05:21:42.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:42.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:42.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:42.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-mon[33554]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T05:21:42.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:42.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:42.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/4178560696' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T05:21:42.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:42 smithi164 ceph-mon[33554]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 5.5 KiB/s wr, 223 op/s 2024-03-26T05:21:43.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:43.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:43.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:43.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[29560]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T05:21:43.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:43.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/4178560696' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[29560]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 5.5 KiB/s wr, 223 op/s 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[38106]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/4178560696' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T05:21:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:42 smithi136 ceph-mon[38106]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 5.5 KiB/s wr, 223 op/s 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 -- 172.21.15.164:0/3719997233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 msgr2=0x7f0528109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 --2- 172.21.15.164:0/3719997233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 0x7f0528109430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f052000b0d0 tx=0x7f052000b3e0 comp rx=0 tx=0).stop 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 -- 172.21.15.164:0/3719997233 shutdown_connections 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 --2- 172.21.15.164:0/3719997233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 0x7f0528109430 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 --2- 172.21.15.164:0/3719997233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0528100fe0 0x7f0528101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 --2- 172.21.15.164:0/3719997233 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05281021e0 0x7f0528102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 -- 172.21.15.164:0/3719997233 >> 172.21.15.164:0/3719997233 conn(0x7f05280fc5a0 msgr2=0x7f05280fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 -- 172.21.15.164:0/3719997233 shutdown_connections 2024-03-26T05:21:43.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.811+0000 7f052e801700 1 -- 172.21.15.164:0/3719997233 wait complete. 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052e801700 1 Processor -- start 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052e801700 1 -- start start 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052e801700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0528100fe0 0x7f05281978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052e801700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05281021e0 0x7f0528197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052e801700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 0x7f052819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052e801700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052819f3e0 con 0x7f0528100fe0 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052e801700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052819f520 con 0x7f0528102b80 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052e801700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052819f6b0 con 0x7f05281021e0 2024-03-26T05:21:43.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052cd9e700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 0x7f052819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052cd9e700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 0x7f052819ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:50206/0 (socket says 172.21.15.164:50206) 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f052cd9e700 1 -- 172.21.15.164:0/1108874311 learned_addr learned my addr 172.21.15.164:0/1108874311 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.812+0000 7f05277fe700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05281021e0 0x7f0528197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f0527fff700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0528100fe0 0x7f05281978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f052cd9e700 1 -- 172.21.15.164:0/1108874311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05281021e0 msgr2=0x7f0528197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f052cd9e700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05281021e0 0x7f0528197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f052cd9e700 1 -- 172.21.15.164:0/1108874311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0528100fe0 msgr2=0x7f05281978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f052cd9e700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0528100fe0 0x7f05281978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f052cd9e700 1 -- 172.21.15.164:0/1108874311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520009d00 con 0x7f0528102b80 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f052cd9e700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 0x7f052819ee10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05200079b0 tx=0x7f05200090d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f05257fa700 1 -- 172.21.15.164:0/1108874311 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052000e040 con 0x7f0528102b80 2024-03-26T05:21:43.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f05257fa700 1 -- 172.21.15.164:0/1108874311 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f0520003c00 con 0x7f0528102b80 2024-03-26T05:21:43.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052819f930 con 0x7f0528102b80 2024-03-26T05:21:43.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f05257fa700 1 -- 172.21.15.164:0/1108874311 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052001b6b0 con 0x7f0528102b80 2024-03-26T05:21:43.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.813+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052819fd40 con 0x7f0528102b80 2024-03-26T05:21:43.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.814+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0528108dc0 con 0x7f0528102b80 2024-03-26T05:21:43.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.815+0000 7f05257fa700 1 -- 172.21.15.164:0/1108874311 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) v1 ==== 91415+0+0 (secure 0 0 0) 0x7f0520005070 con 0x7f0528102b80 2024-03-26T05:21:43.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.816+0000 7f05257fa700 1 --2- 172.21.15.164:0/1108874311 >> v2:172.21.15.136:6800/2365933975 conn(0x7f051006da20 0x7f051006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:43.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.816+0000 7f05257fa700 1 -- 172.21.15.164:0/1108874311 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f052008e640 con 0x7f0528102b80 2024-03-26T05:21:43.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.816+0000 7f0527fff700 1 --2- 172.21.15.164:0/1108874311 >> v2:172.21.15.136:6800/2365933975 conn(0x7f051006da20 0x7f051006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:43.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.816+0000 7f0527fff700 1 --2- 172.21.15.164:0/1108874311 >> v2:172.21.15.136:6800/2365933975 conn(0x7f051006da20 0x7f051006fee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0528102040 tx=0x7f051800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:43.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:43.821+0000 7f05257fa700 1 -- 172.21.15.164:0/1108874311 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0520057f70 con 0x7f0528102b80 2024-03-26T05:21:43.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:43 smithi164 ceph-mon[33554]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T05:21:44.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.047+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi136=a;smithi164=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f05280624c0 con 0x7f051006da20 2024-03-26T05:21:44.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.054+0000 7f05257fa700 1 -- 172.21.15.164:0/1108874311 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f05280624c0 con 0x7f051006da20 2024-03-26T05:21:44.044 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled node-exporter update... 2024-03-26T05:21:44.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 >> v2:172.21.15.136:6800/2365933975 conn(0x7f051006da20 msgr2=0x7f051006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:44.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 --2- 172.21.15.164:0/1108874311 >> v2:172.21.15.136:6800/2365933975 conn(0x7f051006da20 0x7f051006fee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0528102040 tx=0x7f051800a380 comp rx=0 tx=0).stop 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 msgr2=0x7f052819ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 0x7f052819ee10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f05200079b0 tx=0x7f05200090d0 comp rx=0 tx=0).stop 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 shutdown_connections 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 --2- 172.21.15.164:0/1108874311 >> v2:172.21.15.136:6800/2365933975 conn(0x7f051006da20 0x7f051006fee0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0528102b80 0x7f052819ee10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0528100fe0 0x7f05281978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 --2- 172.21.15.164:0/1108874311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05281021e0 0x7f0528197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 >> 172.21.15.164:0/1108874311 conn(0x7f05280fc5a0 msgr2=0x7f05280fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 shutdown_connections 2024-03-26T05:21:44.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:44.057+0000 7f052e801700 1 -- 172.21.15.164:0/1108874311 wait complete. 2024-03-26T05:21:44.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:43 smithi136 ceph-mon[38106]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T05:21:44.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:43 smithi136 ceph-mon[29560]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T05:21:44.811 DEBUG:teuthology.orchestra.run.smithi136:node-exporter.a> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T05:21:44.814 DEBUG:teuthology.orchestra.run.smithi164:node-exporter.b> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T05:21:44.816 INFO:tasks.cephadm:Adding alertmanager.a on smithi136 2024-03-26T05:21:44.817 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi136=a' 2024-03-26T05:21:45.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:44 smithi164 ceph-mon[33554]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T05:21:45.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:44 smithi164 ceph-mon[33554]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi136=a;smithi164=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:45.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:44 smithi164 ceph-mon[33554]: Saving service node-exporter spec with placement smithi136=a;smithi164=b;count:2 2024-03-26T05:21:45.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:44 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:45.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:44 smithi164 ceph-mon[33554]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 5.1 KiB/s wr, 209 op/s 2024-03-26T05:21:45.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[38106]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T05:21:45.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[38106]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi136=a;smithi164=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:45.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[38106]: Saving service node-exporter spec with placement smithi136=a;smithi164=b;count:2 2024-03-26T05:21:45.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:45.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[38106]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 5.1 KiB/s wr, 209 op/s 2024-03-26T05:21:45.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[29560]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T05:21:45.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[29560]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi136=a;smithi164=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:45.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[29560]: Saving service node-exporter spec with placement smithi136=a;smithi164=b;count:2 2024-03-26T05:21:45.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:45.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:44 smithi136 ceph-mon[29560]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 5.1 KiB/s wr, 209 op/s 2024-03-26T05:21:45.436 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 -- 172.21.15.164:0/2613746580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c1021b0 msgr2=0x7fab2c102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 --2- 172.21.15.164:0/2613746580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c1021b0 0x7fab2c102610 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fab200099c0 tx=0x7fab20009cd0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 -- 172.21.15.164:0/2613746580 shutdown_connections 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 --2- 172.21.15.164:0/2613746580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c102b50 0x7fab2c109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 --2- 172.21.15.164:0/2613746580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab2c100fb0 0x7fab2c1013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 --2- 172.21.15.164:0/2613746580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c1021b0 0x7fab2c102610 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 -- 172.21.15.164:0/2613746580 >> 172.21.15.164:0/2613746580 conn(0x7fab2c0fc550 msgr2=0x7fab2c0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 -- 172.21.15.164:0/2613746580 shutdown_connections 2024-03-26T05:21:47.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.287+0000 7fab309ec700 1 -- 172.21.15.164:0/2613746580 wait complete. 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab309ec700 1 Processor -- start 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab309ec700 1 -- start start 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab309ec700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c100fb0 0x7fab2c197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab309ec700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab2c1021b0 0x7fab2c197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab309ec700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c102b50 0x7fab2c19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab309ec700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c19f400 con 0x7fab2c102b50 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab309ec700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c19f540 con 0x7fab2c100fb0 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab309ec700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c19f6d0 con 0x7fab2c1021b0 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2a59c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c100fb0 0x7fab2c197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2a59c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c100fb0 0x7fab2c197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35738/0 (socket says 172.21.15.164:35738) 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2a59c700 1 -- 172.21.15.164:0/3066239301 learned_addr learned my addr 172.21.15.164:0/3066239301 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2ad9d700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c102b50 0x7fab2c19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:47.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab29d9b700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab2c1021b0 0x7fab2c197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:47.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2a59c700 1 -- 172.21.15.164:0/3066239301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab2c1021b0 msgr2=0x7fab2c197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:47.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2a59c700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab2c1021b0 0x7fab2c197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2a59c700 1 -- 172.21.15.164:0/3066239301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c102b50 msgr2=0x7fab2c19ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:47.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2a59c700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c102b50 0x7fab2c19ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.288+0000 7fab2a59c700 1 -- 172.21.15.164:0/3066239301 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab200095f0 con 0x7fab2c100fb0 2024-03-26T05:21:47.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.289+0000 7fab2a59c700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c100fb0 0x7fab2c197870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fab2400b500 tx=0x7fab2400b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:47.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.289+0000 7fab1b7fe700 1 -- 172.21.15.164:0/3066239301 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24011830 con 0x7fab2c100fb0 2024-03-26T05:21:47.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.289+0000 7fab1b7fe700 1 -- 172.21.15.164:0/3066239301 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fab24011e40 con 0x7fab2c100fb0 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.289+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c1034f0 con 0x7fab2c100fb0 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.289+0000 7fab1b7fe700 1 -- 172.21.15.164:0/3066239301 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24010db0 con 0x7fab2c100fb0 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.289+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c103ac0 con 0x7fab2c100fb0 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.290+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c068d40 con 0x7fab2c100fb0 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.290+0000 7fab1b7fe700 1 -- 172.21.15.164:0/3066239301 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) v1 ==== 91415+0+0 (secure 0 0 0) 0x7fab24010380 con 0x7fab2c100fb0 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.291+0000 7fab1b7fe700 1 --2- 172.21.15.164:0/3066239301 >> v2:172.21.15.136:6800/2365933975 conn(0x7fab1406da20 0x7fab1406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.291+0000 7fab1b7fe700 1 -- 172.21.15.164:0/3066239301 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (secure 0 0 0) 0x7fab2408cb50 con 0x7fab2c100fb0 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.291+0000 7fab29d9b700 1 --2- 172.21.15.164:0/3066239301 >> v2:172.21.15.136:6800/2365933975 conn(0x7fab1406da20 0x7fab1406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:47.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.291+0000 7fab29d9b700 1 --2- 172.21.15.164:0/3066239301 >> v2:172.21.15.136:6800/2365933975 conn(0x7fab1406da20 0x7fab1406fee0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab20000c00 tx=0x7fab20009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:47.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.295+0000 7fab1b7fe700 1 -- 172.21.15.164:0/3066239301 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fab24056610 con 0x7fab2c100fb0 2024-03-26T05:21:47.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.502+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fab2c0fe130 con 0x7fab1406da20 2024-03-26T05:21:47.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.510+0000 7fab1b7fe700 1 -- 172.21.15.164:0/3066239301 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fab2c0fe130 con 0x7fab1406da20 2024-03-26T05:21:47.500 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled alertmanager update... 2024-03-26T05:21:47.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 >> v2:172.21.15.136:6800/2365933975 conn(0x7fab1406da20 msgr2=0x7fab1406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:47.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 --2- 172.21.15.164:0/3066239301 >> v2:172.21.15.136:6800/2365933975 conn(0x7fab1406da20 0x7fab1406fee0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab20000c00 tx=0x7fab20009f10 comp rx=0 tx=0).stop 2024-03-26T05:21:47.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c100fb0 msgr2=0x7fab2c197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:47.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c100fb0 0x7fab2c197870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fab2400b500 tx=0x7fab2400b810 comp rx=0 tx=0).stop 2024-03-26T05:21:47.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 shutdown_connections 2024-03-26T05:21:47.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 --2- 172.21.15.164:0/3066239301 >> v2:172.21.15.136:6800/2365933975 conn(0x7fab1406da20 0x7fab1406fee0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab2c100fb0 0x7fab2c197870 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c102b50 0x7fab2c19ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 --2- 172.21.15.164:0/3066239301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab2c1021b0 0x7fab2c197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:47.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 >> 172.21.15.164:0/3066239301 conn(0x7fab2c0fc550 msgr2=0x7fab2c0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:47.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 shutdown_connections 2024-03-26T05:21:47.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:47.512+0000 7fab309ec700 1 -- 172.21.15.164:0/3066239301 wait complete. 2024-03-26T05:21:47.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:47 smithi164 ceph-mon[33554]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 4.2 KiB/s wr, 185 op/s 2024-03-26T05:21:47.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:47.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:47 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:48.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:47 smithi136 ceph-mon[29560]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 4.2 KiB/s wr, 185 op/s 2024-03-26T05:21:48.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:48.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:47 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:48.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:47 smithi136 ceph-mon[38106]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 4.2 KiB/s wr, 185 op/s 2024-03-26T05:21:48.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:48.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:47 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:48.255 DEBUG:teuthology.orchestra.run.smithi136:alertmanager.a> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T05:21:48.258 INFO:tasks.cephadm:Adding grafana.a on smithi164 2024-03-26T05:21:48.258 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi164=a' 2024-03-26T05:21:48.507 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:48.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:48 smithi164 ceph-mon[33554]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:48.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:48 smithi164 ceph-mon[33554]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-03-26T05:21:48.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:48.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:48.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:48.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:48.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:48 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:49.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[38106]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:49.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[38106]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-03-26T05:21:49.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:49.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[29560]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[29560]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:21:49.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:48 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:49 smithi164 ceph-mon[33554]: Deploying daemon prometheus.a on smithi164 2024-03-26T05:21:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:49 smithi164 ceph-mon[33554]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 3.7 KiB/s wr, 162 op/s 2024-03-26T05:21:49.918 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:49 smithi136 ceph-mon[38106]: Deploying daemon prometheus.a on smithi164 2024-03-26T05:21:49.919 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:49 smithi136 ceph-mon[38106]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 3.7 KiB/s wr, 162 op/s 2024-03-26T05:21:49.919 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:49 smithi136 ceph-mon[29560]: Deploying daemon prometheus.a on smithi164 2024-03-26T05:21:49.919 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:49 smithi136 ceph-mon[29560]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 3.7 KiB/s wr, 162 op/s 2024-03-26T05:21:50.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 -- 172.21.15.164:0/183621056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba50103c70 msgr2=0x7fba5010a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 --2- 172.21.15.164:0/183621056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba50103c70 0x7fba5010a520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fba4800b0d0 tx=0x7fba4800b3e0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 -- 172.21.15.164:0/183621056 shutdown_connections 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 --2- 172.21.15.164:0/183621056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba50103c70 0x7fba5010a520 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 --2- 172.21.15.164:0/183621056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba501020d0 0x7fba501024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 --2- 172.21.15.164:0/183621056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba501032d0 0x7fba50103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 -- 172.21.15.164:0/183621056 >> 172.21.15.164:0/183621056 conn(0x7fba500fd690 msgr2=0x7fba500ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 -- 172.21.15.164:0/183621056 shutdown_connections 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 -- 172.21.15.164:0/183621056 wait complete. 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 Processor -- start 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.379+0000 7fba56ca5700 1 -- start start 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba56ca5700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba501020d0 0x7fba501977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:50.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba56ca5700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba501032d0 0x7fba50197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba56ca5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba50103c70 0x7fba5019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba56ca5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5019f360 con 0x7fba50103c70 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba56ca5700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5019f4a0 con 0x7fba501020d0 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba56ca5700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5019f630 con 0x7fba501032d0 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba501020d0 0x7fba501977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba501020d0 0x7fba501977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35756/0 (socket says 172.21.15.164:35756) 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 -- 172.21.15.164:0/912824258 learned_addr learned my addr 172.21.15.164:0/912824258 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba55242700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba50103c70 0x7fba5019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba4ffff700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba501032d0 0x7fba50197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 -- 172.21.15.164:0/912824258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba501032d0 msgr2=0x7fba50197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba501032d0 0x7fba50197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 -- 172.21.15.164:0/912824258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba50103c70 msgr2=0x7fba5019ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba50103c70 0x7fba5019ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 -- 172.21.15.164:0/912824258 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba48009d00 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba54a41700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba501020d0 0x7fba501977d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba4400d980 tx=0x7fba4400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.380+0000 7fba4dffb700 1 -- 172.21.15.164:0/912824258 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44004020 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.381+0000 7fba4dffb700 1 -- 172.21.15.164:0/912824258 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fba4400f690 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.381+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5019f910 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.381+0000 7fba4dffb700 1 -- 172.21.15.164:0/912824258 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44003a90 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.381+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5019fe10 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.382+0000 7fba4dffb700 1 -- 172.21.15.164:0/912824258 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) v1 ==== 91415+0+0 (secure 0 0 0) 0x7fba4400fb00 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.383+0000 7fba4dffb700 1 --2- 172.21.15.164:0/912824258 >> v2:172.21.15.136:6800/2365933975 conn(0x7fba4006da20 0x7fba4006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.383+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba50109430 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.383+0000 7fba4dffb700 1 -- 172.21.15.164:0/912824258 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (secure 0 0 0) 0x7fba4408ce20 con 0x7fba501020d0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.383+0000 7fba4ffff700 1 --2- 172.21.15.164:0/912824258 >> v2:172.21.15.136:6800/2365933975 conn(0x7fba4006da20 0x7fba4006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.383+0000 7fba4ffff700 1 --2- 172.21.15.164:0/912824258 >> v2:172.21.15.136:6800/2365933975 conn(0x7fba4006da20 0x7fba4006fee0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009c60 tx=0x7fba3c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:50.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.388+0000 7fba4dffb700 1 -- 172.21.15.164:0/912824258 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fba44056750 con 0x7fba501020d0 2024-03-26T05:21:50.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.656+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 --> v2:172.21.15.136:6800/2365933975 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fba5010d3f0 con 0x7fba4006da20 2024-03-26T05:21:50.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.662+0000 7fba4dffb700 1 -- 172.21.15.164:0/912824258 <== mgr.14150 v2:172.21.15.136:6800/2365933975 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fba5010d3f0 con 0x7fba4006da20 2024-03-26T05:21:50.652 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled grafana update... 2024-03-26T05:21:50.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 >> v2:172.21.15.136:6800/2365933975 conn(0x7fba4006da20 msgr2=0x7fba4006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:50.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 --2- 172.21.15.164:0/912824258 >> v2:172.21.15.136:6800/2365933975 conn(0x7fba4006da20 0x7fba4006fee0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009c60 tx=0x7fba3c009260 comp rx=0 tx=0).stop 2024-03-26T05:21:50.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba501020d0 msgr2=0x7fba501977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:50.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba501020d0 0x7fba501977d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba4400d980 tx=0x7fba4400dc90 comp rx=0 tx=0).stop 2024-03-26T05:21:50.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 shutdown_connections 2024-03-26T05:21:50.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 --2- 172.21.15.164:0/912824258 >> v2:172.21.15.136:6800/2365933975 conn(0x7fba4006da20 0x7fba4006fee0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba501020d0 0x7fba501977d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba50103c70 0x7fba5019ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 --2- 172.21.15.164:0/912824258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba501032d0 0x7fba50197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:50.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 >> 172.21.15.164:0/912824258 conn(0x7fba500fd690 msgr2=0x7fba50106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:50.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.664+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 shutdown_connections 2024-03-26T05:21:50.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:21:50.665+0000 7fba56ca5700 1 -- 172.21.15.164:0/912824258 wait complete. 2024-03-26T05:21:51.748 DEBUG:teuthology.orchestra.run.smithi164:grafana.a> sudo journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T05:21:51.750 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T05:21:51.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T05:21:51.967 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:21:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:21:51.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:51 smithi164 ceph-mon[33554]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1.1 KiB/s wr, 52 op/s 2024-03-26T05:21:51.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:51 smithi164 ceph-mon[33554]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:51.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:51 smithi164 ceph-mon[33554]: Saving service grafana spec with placement smithi164=a;count:1 2024-03-26T05:21:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:51 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:51 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:52.000 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:21:52.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[38106]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1.1 KiB/s wr, 52 op/s 2024-03-26T05:21:52.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[38106]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:52.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[38106]: Saving service grafana spec with placement smithi164=a;count:1 2024-03-26T05:21:52.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:52.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[29560]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1.1 KiB/s wr, 52 op/s 2024-03-26T05:21:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[29560]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T05:21:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[29560]: Saving service grafana spec with placement smithi164=a;count:1 2024-03-26T05:21:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:51 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:21:53.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 -- 172.21.15.136:0/3497879418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 msgr2=0x7f121c109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:53.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 --2- 172.21.15.136:0/3497879418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 0x7f121c109410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1214009920 tx=0x7f1214009c30 comp rx=0 tx=0).stop 2024-03-26T05:21:53.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 -- 172.21.15.136:0/3497879418 shutdown_connections 2024-03-26T05:21:53.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 --2- 172.21.15.136:0/3497879418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 0x7f121c109410 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 --2- 172.21.15.136:0/3497879418 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121c100fc0 0x7f121c1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 --2- 172.21.15.136:0/3497879418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f121c1021c0 0x7f121c102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 -- 172.21.15.136:0/3497879418 >> 172.21.15.136:0/3497879418 conn(0x7f121c0fc580 msgr2=0x7f121c0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:53.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 -- 172.21.15.136:0/3497879418 shutdown_connections 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.686+0000 7f122207c700 1 -- 172.21.15.136:0/3497879418 wait complete. 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f122207c700 1 Processor -- start 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f122207c700 1 -- start start 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f122207c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121c100fc0 0x7f121c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f122207c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f121c1021c0 0x7f121c199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f122207c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 0x7f121c1a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f122207c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1a1430 con 0x7f121c102b60 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f122207c700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1a1570 con 0x7f121c1021c0 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f122207c700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1a1700 con 0x7f121c100fc0 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f121b7fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121c100fc0 0x7f121c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f121bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 0x7f121c1a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f121bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 0x7f121c1a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:50698/0 (socket says 172.21.15.136:50698) 2024-03-26T05:21:53.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f121bfff700 1 -- 172.21.15.136:0/2798569499 learned_addr learned my addr 172.21.15.136:0/2798569499 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f121b7fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121c100fc0 0x7f121c1998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:45894/0 (socket says 172.21.15.136:45894) 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.687+0000 7f121affd700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f121c1021c0 0x7f121c199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f121b7fe700 1 -- 172.21.15.136:0/2798569499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f121c1021c0 msgr2=0x7f121c199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f121b7fe700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f121c1021c0 0x7f121c199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f121b7fe700 1 -- 172.21.15.136:0/2798569499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 msgr2=0x7f121c1a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f121b7fe700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 0x7f121c1a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f121b7fe700 1 -- 172.21.15.136:0/2798569499 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c009d00 con 0x7f121c100fc0 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f121bfff700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 0x7f121c1a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f121b7fe700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121c100fc0 0x7f121c1998a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f120c00c340 tx=0x7f120c00c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f1218ff9700 1 -- 172.21.15.136:0/2798569499 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c00e050 con 0x7f121c100fc0 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f1218ff9700 1 -- 172.21.15.136:0/2798569499 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f120c00f040 con 0x7f121c100fc0 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f1218ff9700 1 -- 172.21.15.136:0/2798569499 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c0079d0 con 0x7f121c100fc0 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1214009520 con 0x7f121c100fc0 2024-03-26T05:21:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.688+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121c1a1d00 con 0x7f121c100fc0 2024-03-26T05:21:53.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.689+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121c106b90 con 0x7f121c100fc0 2024-03-26T05:21:53.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.689+0000 7f1218ff9700 1 -- 172.21.15.136:0/2798569499 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 16) v1 ==== 91415+0+0 (secure 0 0 0) 0x7f120c003b70 con 0x7f121c100fc0 2024-03-26T05:21:53.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.690+0000 7f1218ff9700 1 --2- 172.21.15.136:0/2798569499 >> v2:172.21.15.136:6800/2365933975 conn(0x7f120406dac0 0x7f120406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:21:53.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.690+0000 7f121affd700 1 --2- 172.21.15.136:0/2798569499 >> v2:172.21.15.136:6800/2365933975 conn(0x7f120406dac0 0x7f120406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:21:53.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.690+0000 7f1218ff9700 1 -- 172.21.15.136:0/2798569499 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f120c08cc60 con 0x7f121c100fc0 2024-03-26T05:21:53.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.690+0000 7f121affd700 1 --2- 172.21.15.136:0/2798569499 >> v2:172.21.15.136:6800/2365933975 conn(0x7f120406dac0 0x7f120406ff80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1210009c60 tx=0x7f1210009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:21:53.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.695+0000 7f1218ff9700 1 -- 172.21.15.136:0/2798569499 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f120c0566c0 con 0x7f121c100fc0 2024-03-26T05:21:53.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:53.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:53 smithi136 ceph-mon[38106]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-26T05:21:53.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:53.823 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:53 smithi136 ceph-mon[29560]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-26T05:21:53.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.968+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f121c10c1e0 con 0x7f121c100fc0 2024-03-26T05:21:53.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.972+0000 7f1218ff9700 1 -- 172.21.15.136:0/2798569499 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f120c059d80 con 0x7f121c100fc0 2024-03-26T05:21:53.962 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-03-26T05:21:53.963 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBxWwJmbbfYORAAOyDKbld/I2LDVl2DNEeB4A== 2024-03-26T05:21:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 >> v2:172.21.15.136:6800/2365933975 conn(0x7f120406dac0 msgr2=0x7f120406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 --2- 172.21.15.136:0/2798569499 >> v2:172.21.15.136:6800/2365933975 conn(0x7f120406dac0 0x7f120406ff80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1210009c60 tx=0x7f1210009260 comp rx=0 tx=0).stop 2024-03-26T05:21:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121c100fc0 msgr2=0x7f121c1998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:21:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121c100fc0 0x7f121c1998a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f120c00c340 tx=0x7f120c00c650 comp rx=0 tx=0).stop 2024-03-26T05:21:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 shutdown_connections 2024-03-26T05:21:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 --2- 172.21.15.136:0/2798569499 >> v2:172.21.15.136:6800/2365933975 conn(0x7f120406dac0 0x7f120406ff80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f121c102b60 0x7f121c1a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121c100fc0 0x7f121c1998a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 --2- 172.21.15.136:0/2798569499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f121c1021c0 0x7f121c199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:21:53.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 >> 172.21.15.136:0/2798569499 conn(0x7f121c0fc580 msgr2=0x7f121c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:21:53.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 shutdown_connections 2024-03-26T05:21:53.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:21:53.975+0000 7f122207c700 1 -- 172.21.15.136:0/2798569499 wait complete. 2024-03-26T05:21:53.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:21:53.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:53 smithi164 ceph-mon[33554]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-26T05:21:54.633 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:21:54.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T05:21:54.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T05:21:54.679 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T05:21:54.899 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2798569499' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:21:54.899 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:21:54.899 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:21:54.899 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2798569499' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:21:54.899 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:21:54.899 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:21:54.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2798569499' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:21:54.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:21:54.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:21:55.788 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T05:21:55.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:55 smithi164 ceph-mon[33554]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-26T05:21:56.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:55 smithi136 ceph-mon[29560]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-26T05:21:56.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:55 smithi136 ceph-mon[38106]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-26T05:21:56.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T05:21:56.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-26T05:21:56.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:56 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T05:21:57.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T05:21:57.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-26T05:21:57.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:56 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T05:21:57.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T05:21:57.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-26T05:21:57.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:56 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T05:21:57.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:57 smithi164 ceph-mon[33554]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:21:57.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T05:21:57.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-26T05:21:57.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:57 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-26T05:21:57.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:57 smithi164 ceph-mon[33554]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T05:21:58.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[38106]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:21:58.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T05:21:58.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-26T05:21:58.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-26T05:21:58.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[38106]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T05:21:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[29560]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:21:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T05:21:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-26T05:21:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-26T05:21:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:57 smithi136 ceph-mon[29560]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T05:21:58.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:58 smithi164 ceph-mon[33554]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T05:21:59.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:58 smithi136 ceph-mon[38106]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T05:21:59.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:58 smithi136 ceph-mon[29560]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T05:21:59.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:21:59 smithi164 ceph-mon[33554]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:00.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:21:59 smithi136 ceph-mon[38106]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:00.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:21:59 smithi136 ceph-mon[29560]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 -- 172.21.15.164:0/2739109448 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154100fd0 msgr2=0x7fa1541013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 --2- 172.21.15.164:0/2739109448 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154100fd0 0x7fa1541013f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa1500099c0 tx=0x7fa150009cd0 comp rx=0 tx=0).stop 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 -- 172.21.15.164:0/2739109448 shutdown_connections 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 --2- 172.21.15.164:0/2739109448 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154100fd0 0x7fa1541013f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 --2- 172.21.15.164:0/2739109448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa154102b70 0x7fa154109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 --2- 172.21.15.164:0/2739109448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1541021d0 0x7fa154102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 -- 172.21.15.164:0/2739109448 >> 172.21.15.164:0/2739109448 conn(0x7fa1540fc590 msgr2=0x7fa1540fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 -- 172.21.15.164:0/2739109448 shutdown_connections 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.901+0000 7fa15bc33700 1 -- 172.21.15.164:0/2739109448 wait complete. 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15bc33700 1 Processor -- start 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15bc33700 1 -- start start 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15bc33700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa154100fd0 0x7fa1541965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15bc33700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1541021d0 0x7fa154196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15bc33700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154102b70 0x7fa15419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15bc33700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15419e170 con 0x7fa154100fd0 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15bc33700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15406baa0 con 0x7fa154102b70 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15bc33700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15406bc60 con 0x7fa1541021d0 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15a1d0700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154102b70 0x7fa15419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa1599cf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa154100fd0 0x7fa1541965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15a1d0700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154102b70 0x7fa15419dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:53894/0 (socket says 172.21.15.164:53894) 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa15a1d0700 1 -- 172.21.15.164:0/3686550068 learned_addr learned my addr 172.21.15.164:0/3686550068 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa1591ce700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1541021d0 0x7fa154196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa1591ce700 1 -- 172.21.15.164:0/3686550068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154102b70 msgr2=0x7fa15419dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa1591ce700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154102b70 0x7fa15419dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa1591ce700 1 -- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa154100fd0 msgr2=0x7fa1541965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa1591ce700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa154100fd0 0x7fa1541965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.902+0000 7fa1591ce700 1 -- 172.21.15.164:0/3686550068 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1500095f0 con 0x7fa1541021d0 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.903+0000 7fa1599cf700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa154100fd0 0x7fa1541965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.903+0000 7fa1591ce700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1541021d0 0x7fa154196b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa14400d950 tx=0x7fa14400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.903+0000 7fa14affd700 1 -- 172.21.15.164:0/3686550068 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14400f890 con 0x7fa1541021d0 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.903+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15406a780 con 0x7fa1541021d0 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.903+0000 7fa14affd700 1 -- 172.21.15.164:0/3686550068 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa144010460 con 0x7fa1541021d0 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.903+0000 7fa14affd700 1 -- 172.21.15.164:0/3686550068 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14400fea0 con 0x7fa1541021d0 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.903+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15406acd0 con 0x7fa1541021d0 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.904+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa154050590 con 0x7fa1541021d0 2024-03-26T05:22:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.905+0000 7fa14affd700 1 -- 172.21.15.164:0/3686550068 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 16) v1 ==== 91415+0+0 (secure 0 0 0) 0x7fa14400f9f0 con 0x7fa1541021d0 2024-03-26T05:22:00.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:00 smithi164 ceph-mon[33554]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T05:22:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.905+0000 7fa14affd700 1 --2- 172.21.15.164:0/3686550068 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa14006dac0 0x7fa14006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.905+0000 7fa14affd700 1 -- 172.21.15.164:0/3686550068 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fa14408cf60 con 0x7fa1541021d0 2024-03-26T05:22:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.905+0000 7fa1599cf700 1 --2- 172.21.15.164:0/3686550068 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa14006dac0 0x7fa14006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.906+0000 7fa1599cf700 1 --2- 172.21.15.164:0/3686550068 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa14006dac0 0x7fa14006ff80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa150006010 tx=0x7fa150005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:00.911+0000 7fa14affd700 1 -- 172.21.15.164:0/3686550068 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa1440c1280 con 0x7fa1541021d0 2024-03-26T05:22:01.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:00 smithi136 ceph-mon[29560]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T05:22:01.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:00 smithi136 ceph-mon[38106]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T05:22:01.162 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 podman[61650]: 2024-03-26 05:22:00.981009238 +0000 UTC m=+0.019136557 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T05:22:01.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.203+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa15410c1c0 con 0x7fa1541021d0 2024-03-26T05:22:01.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.208+0000 7fa14affd700 1 -- 172.21.15.164:0/3686550068 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa144056850 con 0x7fa1541021d0 2024-03-26T05:22:01.199 INFO:teuthology.orchestra.run.smithi164.stdout:[client.1] 2024-03-26T05:22:01.199 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB5WwJmxmdIDBAAXearuMrXtGXVMRSs7wBMkA== 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa14006dac0 msgr2=0x7fa14006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 --2- 172.21.15.164:0/3686550068 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa14006dac0 0x7fa14006ff80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa150006010 tx=0x7fa150005e80 comp rx=0 tx=0).stop 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1541021d0 msgr2=0x7fa154196b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1541021d0 0x7fa154196b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa14400d950 tx=0x7fa14400dc60 comp rx=0 tx=0).stop 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 shutdown_connections 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 --2- 172.21.15.164:0/3686550068 >> v2:172.21.15.136:6800/2365933975 conn(0x7fa14006dac0 0x7fa14006ff80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa154102b70 0x7fa15419dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa154100fd0 0x7fa1541965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 --2- 172.21.15.164:0/3686550068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1541021d0 0x7fa154196b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 >> 172.21.15.164:0/3686550068 conn(0x7fa1540fc590 msgr2=0x7fa1540fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 shutdown_connections 2024-03-26T05:22:01.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-26T05:22:01.211+0000 7fa15bc33700 1 -- 172.21.15.164:0/3686550068 wait complete. 2024-03-26T05:22:01.468 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 podman[61650]: 2024-03-26 05:22:01.303760047 +0000 UTC m=+0.341887353 container init 593a7e65c964633e7b2e112e0404e8d55ef4a0c82bfa53f9dd07e5dffce5c1ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:22:01.468 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 podman[61650]: 2024-03-26 05:22:01.306512099 +0000 UTC m=+0.344639384 container start 593a7e65c964633e7b2e112e0404e8d55ef4a0c82bfa53f9dd07e5dffce5c1ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:22:01.468 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.329Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T05:22:01.468 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.329Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T05:22:01.468 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.329Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi164 (none))" 2024-03-26T05:22:01.468 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.329Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T05:22:01.468 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.329Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.331Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.331Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.333Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.333Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.335Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.335Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.49µs 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.335Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.335Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.335Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.11µs wal_replay_duration=200.902µs wbl_replay_duration=104ns total_replay_duration=242.34µs 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.336Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.336Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.336Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.352Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.35938ms db_storage=768ns remote_storage=1.081µs web_handler=196ns query_engine=492ns scrape=384.816µs scrape_sd=58.762µs notify=704ns notify_sd=1.649µs rules=15.689398ms tracing=6.108µs 2024-03-26T05:22:01.469 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.352Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T05:22:01.470 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:22:01.352Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T05:22:01.967 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:22:01.967 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 bash[61650]: 593a7e65c964633e7b2e112e0404e8d55ef4a0c82bfa53f9dd07e5dffce5c1ed 2024-03-26T05:22:01.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:22:01 smithi164 systemd[1]: Started Ceph prometheus.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:22:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.164:0/3686550068' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:22:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:22:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:22:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:01 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T05:22:02.055 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T05:22:02.055 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T05:22:02.055 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T05:22:02.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.164:0/3686550068' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:22:02.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:22:02.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:22:02.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:02.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:02.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:02.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T05:22:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.164:0/3686550068' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:22:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T05:22:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T05:22:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' 2024-03-26T05:22:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:01 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T05:22:02.098 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T05:22:02.099 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T05:22:02.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T05:22:02.349 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:02.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:02 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:02.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:02 smithi136 ceph-mon[38106]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T05:22:02.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:02 smithi136 ceph-mon[38106]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T05:22:02.733 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:02 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:02.733 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:02 smithi136 ceph-mon[29560]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T05:22:02.733 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:02 smithi136 ceph-mon[29560]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T05:22:02.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:02 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:02.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:02 smithi164 ceph-mon[33554]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T05:22:02.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:02 smithi164 ceph-mon[33554]: from='mgr.14150 172.21.15.136:0/2732091597' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T05:22:02.968 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: ignoring --setuser ceph since I am not root 2024-03-26T05:22:02.968 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: ignoring --setgroup ceph since I am not root 2024-03-26T05:22:02.968 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:02 smithi164 ceph-mgr[35442]: -- 172.21.15.164:0/2883967900 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1bff4a1a0 con 0x55f1bff20c00 2024-03-26T05:22:03.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:02 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ignoring --setuser ceph since I am not root 2024-03-26T05:22:03.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:02 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ignoring --setgroup ceph since I am not root 2024-03-26T05:22:03.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:03 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:03.290+0000 7f21cd298200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T05:22:03.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:03 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:03.383+0000 7f21cd298200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T05:22:03.717 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:03.271+0000 7f1084c1d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T05:22:03.718 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:03.364+0000 7f1084c1d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T05:22:03.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:03 smithi136 ceph-mon[38106]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T05:22:03.937 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:03 smithi136 ceph-mon[29560]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T05:22:03.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:03 smithi164 ceph-mon[33554]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T05:22:04.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 -- 172.21.15.136:0/2729932098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec102390 msgr2=0x7fdcec105c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:04.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 --2- 172.21.15.136:0/2729932098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec102390 0x7fdcec105c20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009990 tx=0x7fdcdc009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 -- 172.21.15.136:0/2729932098 shutdown_connections 2024-03-26T05:22:04.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 --2- 172.21.15.136:0/2729932098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcec1019e0 0x7fdcec101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 --2- 172.21.15.136:0/2729932098 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdcec106300 0x7fdcec10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 --2- 172.21.15.136:0/2729932098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec102390 0x7fdcec105c20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 -- 172.21.15.136:0/2729932098 >> 172.21.15.136:0/2729932098 conn(0x7fdcec077580 msgr2=0x7fdcec0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 -- 172.21.15.136:0/2729932098 shutdown_connections 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.181+0000 7fdcf426f700 1 -- 172.21.15.136:0/2729932098 wait complete. 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf426f700 1 Processor -- start 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf426f700 1 -- start start 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf426f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec1019e0 0x7fdcec19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf426f700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcec102390 0x7fdcec19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf426f700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdcec106300 0x7fdcec196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf426f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec197120 con 0x7fdcec1019e0 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf426f700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec197260 con 0x7fdcec102390 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf426f700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec197420 con 0x7fdcec106300 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf280c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdcec106300 0x7fdcec196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:04.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf280c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdcec106300 0x7fdcec196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:58576/0 (socket says 172.21.15.136:58576) 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf280c700 1 -- 172.21.15.136:0/4083179270 learned_addr learned my addr 172.21.15.136:0/4083179270 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf200b700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec1019e0 0x7fdcec19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.182+0000 7fdcf180a700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcec102390 0x7fdcec19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdcf200b700 1 -- 172.21.15.136:0/4083179270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdcec106300 msgr2=0x7fdcec196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdcf200b700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdcec106300 0x7fdcec196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdcf200b700 1 -- 172.21.15.136:0/4083179270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcec102390 msgr2=0x7fdcec19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdcf200b700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcec102390 0x7fdcec19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdcf200b700 1 -- 172.21.15.136:0/4083179270 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4009c40 con 0x7fdcec1019e0 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdcf200b700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec1019e0 0x7fdcec19c900 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdce800d950 tx=0x7fdce800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdce37fe700 1 -- 172.21.15.136:0/4083179270 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce800f890 con 0x7fdcec1019e0 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc0095f0 con 0x7fdcec1019e0 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdce37fe700 1 -- 172.21.15.136:0/4083179270 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fdce8010460 con 0x7fdcec1019e0 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdce37fe700 1 -- 172.21.15.136:0/4083179270 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce800fea0 con 0x7fdcec1019e0 2024-03-26T05:22:04.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.183+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec100670 con 0x7fdcec1019e0 2024-03-26T05:22:04.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.184+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec068d40 con 0x7fdcec1019e0 2024-03-26T05:22:04.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.185+0000 7fdce37fe700 1 -- 172.21.15.136:0/4083179270 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 91429+0+0 (secure 0 0 0) 0x7fdce800f9f0 con 0x7fdcec1019e0 2024-03-26T05:22:04.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.185+0000 7fdce37fe700 1 --2- 172.21.15.136:0/4083179270 >> v2:172.21.15.136:6800/2365933975 conn(0x7fdcd806da70 0x7fdcd806ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:04.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.185+0000 7fdcf180a700 1 -- 172.21.15.136:0/4083179270 >> v2:172.21.15.136:6800/2365933975 conn(0x7fdcd806da70 msgr2=0x7fdcd806ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/2365933975 2024-03-26T05:22:04.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.185+0000 7fdce37fe700 1 -- 172.21.15.136:0/4083179270 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fdce808cfb0 con 0x7fdcec1019e0 2024-03-26T05:22:04.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.185+0000 7fdcf180a700 1 --2- 172.21.15.136:0/4083179270 >> v2:172.21.15.136:6800/2365933975 conn(0x7fdcd806da70 0x7fdcd806ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T05:22:04.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.190+0000 7fdce37fe700 1 -- 172.21.15.136:0/4083179270 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdce80568a0 con 0x7fdcec1019e0 2024-03-26T05:22:04.271 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:04.037+0000 7f21cd298200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T05:22:04.271 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:04.132+0000 7f21cd298200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T05:22:04.271 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:04.280+0000 7f21cd298200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T05:22:04.318 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:03.999+0000 7f1084c1d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T05:22:04.318 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:04.092+0000 7f1084c1d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T05:22:04.318 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:04.233+0000 7f1084c1d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T05:22:04.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.386+0000 7fdcf180a700 1 -- 172.21.15.136:0/4083179270 >> v2:172.21.15.136:6800/2365933975 conn(0x7fdcd806da70 msgr2=0x7fdcd806ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/2365933975 2024-03-26T05:22:04.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.386+0000 7fdcf180a700 1 --2- 172.21.15.136:0/4083179270 >> v2:172.21.15.136:6800/2365933975 conn(0x7fdcd806da70 0x7fdcd806ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T05:22:04.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.447+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fdcec102770 con 0x7fdcec1019e0 2024-03-26T05:22:04.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.450+0000 7fdce37fe700 1 -- 172.21.15.136:0/4083179270 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+174808 (secure 0 0 0) 0x7fdce8056030 con 0x7fdcec1019e0 2024-03-26T05:22:04.444 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 >> v2:172.21.15.136:6800/2365933975 conn(0x7fdcd806da70 msgr2=0x7fdcd806ff30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 --2- 172.21.15.136:0/4083179270 >> v2:172.21.15.136:6800/2365933975 conn(0x7fdcd806da70 0x7fdcd806ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec1019e0 msgr2=0x7fdcec19c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec1019e0 0x7fdcec19c900 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdce800d950 tx=0x7fdce800dc60 comp rx=0 tx=0).stop 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 shutdown_connections 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 --2- 172.21.15.136:0/4083179270 >> v2:172.21.15.136:6800/2365933975 conn(0x7fdcd806da70 0x7fdcd806ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdcec106300 0x7fdcec196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcec1019e0 0x7fdcec19c900 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 --2- 172.21.15.136:0/4083179270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcec102390 0x7fdcec19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:04.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 >> 172.21.15.136:0/4083179270 conn(0x7fdcec077580 msgr2=0x7fdcec103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:04.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 shutdown_connections 2024-03-26T05:22:04.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:04.453+0000 7fdcf426f700 1 -- 172.21.15.136:0/4083179270 wait complete. 2024-03-26T05:22:04.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:04.378+0000 7f21cd298200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T05:22:04.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:04.464+0000 7f21cd298200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T05:22:04.661 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:04.327+0000 7f1084c1d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T05:22:04.662 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:04.410+0000 7f1084c1d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T05:22:04.881 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4083179270' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T05:22:04.882 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4083179270' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T05:22:04.882 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:04.734+0000 7f21cd298200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T05:22:04.952 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4083179270' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T05:22:04.952 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:04.670+0000 7f1084c1d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T05:22:04.952 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:04.780+0000 7f1084c1d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T05:22:04.952 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:04.867+0000 7f1084c1d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T05:22:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:04.844+0000 7f21cd298200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T05:22:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:04 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:04.936+0000 7f21cd298200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T05:22:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:05 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:05.029+0000 7f21cd298200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T05:22:05.217 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:04 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:04.961+0000 7f1084c1d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T05:22:05.217 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:05 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:05.136+0000 7f1084c1d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T05:22:05.409 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2365933975}]},"active_addr":"172.21.15.136:6800/2365933975","active_change":"2024-03-26T05:16:56.499172+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24118,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.136:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":28828828}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":1070436131}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":1158073833}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":452904540}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":2667793025}]}]} 2024-03-26T05:22:05.414 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T05:22:05.414 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T05:22:05.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T05:22:05.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:05 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:05.206+0000 7f21cd298200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T05:22:05.688 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:06.350 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:06.230+0000 7f21cd298200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T05:22:06.350 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:06.359+0000 7f21cd298200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T05:22:06.467 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:06 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:06.094+0000 7f1084c1d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T05:22:06.467 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:06 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:06.226+0000 7f1084c1d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T05:22:06.688 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:06.698+0000 7f21cd298200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T05:22:06.844 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:06 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:06.561+0000 7f1084c1d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T05:22:06.845 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:06 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:06.653+0000 7f1084c1d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T05:22:06.845 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:06 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:06.853+0000 7f1084c1d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:22:06.979 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:06.788+0000 7f21cd298200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T05:22:06.979 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:06.989+0000 7f21cd298200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:22:07.218 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:07 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:07.042+0000 7f1084c1d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T05:22:07.301 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:07.185+0000 7f21cd298200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T05:22:07.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 -- 172.21.15.136:0/3800630079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c102c70 msgr2=0x7f4c6c106cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:07.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 --2- 172.21.15.136:0/3800630079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c102c70 0x7f4c6c106cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0098f0 tx=0x7f4c5c009c00 comp rx=0 tx=0).stop 2024-03-26T05:22:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 -- 172.21.15.136:0/3800630079 shutdown_connections 2024-03-26T05:22:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 --2- 172.21.15.136:0/3800630079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6c1073a0 0x7f4c6c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 --2- 172.21.15.136:0/3800630079 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c6c1022c0 0x7f4c6c1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 --2- 172.21.15.136:0/3800630079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c102c70 0x7f4c6c106cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 -- 172.21.15.136:0/3800630079 >> 172.21.15.136:0/3800630079 conn(0x7f4c6c0fdb30 msgr2=0x7f4c6c0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 -- 172.21.15.136:0/3800630079 shutdown_connections 2024-03-26T05:22:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.549+0000 7f4c70b50700 1 -- 172.21.15.136:0/3800630079 wait complete. 2024-03-26T05:22:07.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c70b50700 1 Processor -- start 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c70b50700 1 -- start start 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c70b50700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c1022c0 0x7f4c6c197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c70b50700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6c102c70 0x7f4c6c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c70b50700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c6c1073a0 0x7f4c6c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c70b50700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c19cc70 con 0x7f4c6c1022c0 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c70b50700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c19cde0 con 0x7f4c6c102c70 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c70b50700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c19cfa0 con 0x7f4c6c1073a0 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c6a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c1022c0 0x7f4c6c197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c6ad9d700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c6c1073a0 0x7f4c6c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c6a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c1022c0 0x7f4c6c197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55620/0 (socket says 172.21.15.136:55620) 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c6ad9d700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c6c1073a0 0x7f4c6c19c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:58594/0 (socket says 172.21.15.136:58594) 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c6a59c700 1 -- 172.21.15.136:0/257942043 learned_addr learned my addr 172.21.15.136:0/257942043 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:07.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.550+0000 7f4c69d9b700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6c102c70 0x7f4c6c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c6a59c700 1 -- 172.21.15.136:0/257942043 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c6c1073a0 msgr2=0x7f4c6c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c6a59c700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c6c1073a0 0x7f4c6c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c6a59c700 1 -- 172.21.15.136:0/257942043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6c102c70 msgr2=0x7f4c6c1982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c6a59c700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6c102c70 0x7f4c6c1982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c6a59c700 1 -- 172.21.15.136:0/257942043 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c600095f0 con 0x7f4c6c1022c0 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c6ad9d700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c6c1073a0 0x7f4c6c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c69d9b700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6c102c70 0x7f4c6c1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c6a59c700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c1022c0 0x7f4c6c197d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000ec10 tx=0x7f4c6000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c5b7fe700 1 -- 172.21.15.136:0/257942043 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000cc90 con 0x7f4c6c1022c0 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c009520 con 0x7f4c6c1022c0 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c5b7fe700 1 -- 172.21.15.136:0/257942043 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f4c600044d0 con 0x7f4c6c1022c0 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c5b7fe700 1 -- 172.21.15.136:0/257942043 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60010430 con 0x7f4c6c1022c0 2024-03-26T05:22:07.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.551+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c1a4910 con 0x7f4c6c1022c0 2024-03-26T05:22:07.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.552+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c005120 con 0x7f4c6c1022c0 2024-03-26T05:22:07.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.553+0000 7f4c5b7fe700 1 -- 172.21.15.136:0/257942043 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 91429+0+0 (secure 0 0 0) 0x7f4c60010590 con 0x7f4c6c1022c0 2024-03-26T05:22:07.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.553+0000 7f4c5b7fe700 1 --2- 172.21.15.136:0/257942043 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4c5406da70 0x7f4c5406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:07.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.553+0000 7f4c69d9b700 1 -- 172.21.15.136:0/257942043 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4c5406da70 msgr2=0x7f4c5406ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/2365933975 2024-03-26T05:22:07.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.553+0000 7f4c5b7fe700 1 -- 172.21.15.136:0/257942043 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f4c60014070 con 0x7f4c6c1022c0 2024-03-26T05:22:07.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.553+0000 7f4c69d9b700 1 --2- 172.21.15.136:0/257942043 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4c5406da70 0x7f4c5406ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T05:22:07.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.558+0000 7f4c5b7fe700 1 -- 172.21.15.136:0/257942043 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4c60056fe0 con 0x7f4c6c1022c0 2024-03-26T05:22:07.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:07.479+0000 7f21cd298200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:22:07.584 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:07 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:07.333+0000 7f1084c1d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T05:22:07.585 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:07 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:07.594+0000 7f1084c1d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T05:22:07.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.754+0000 7f4c69d9b700 1 -- 172.21.15.136:0/257942043 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4c5406da70 msgr2=0x7f4c5406ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.136:6800/2365933975 2024-03-26T05:22:07.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.754+0000 7f4c69d9b700 1 --2- 172.21.15.136:0/257942043 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4c5406da70 0x7f4c5406ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T05:22:07.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.760+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c4c0056a0 con 0x7f4c6c1022c0 2024-03-26T05:22:07.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.761+0000 7f4c5b7fe700 1 -- 172.21.15.136:0/257942043 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+20695 (secure 0 0 0) 0x7f4c6005a6a0 con 0x7f4c6c1022c0 2024-03-26T05:22:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:22:07.753 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":71,"fsid":"d69b39a8-eb2f-11ee-95d0-87774f69a715","created":"2024-03-26T05:15:55.712909+0000","modified":"2024-03-26T05:21:57.674500+0000","last_up_change":"2024-03-26T05:21:17.162399+0000","last_in_change":"2024-03-26T05:20:59.552150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T05:19:14.832191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T05:21:28.746312+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T05:21:29.507514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T05:21:31.156582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T05:21:33.087723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T05:21:35.140790+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4698473-d228-4853-b404-54b255f0a9d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":1923051194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1923051194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1923051194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":1923051194}]},"public_addr":"172.21.15.136:6801/1923051194","cluster_addr":"172.21.15.136:6802/1923051194","heartbeat_back_addr":"172.21.15.136:6804/1923051194","heartbeat_front_addr":"172.21.15.136:6803/1923051194","state":["exists","up"]},{"osd":1,"uuid":"800f61d3-4725-45b6-af8c-0653cb8b4c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":1223127568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1223127568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1223127568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":1223127568}]},"public_addr":"172.21.15.136:6805/1223127568","cluster_addr":"172.21.15.136:6806/1223127568","heartbeat_back_addr":"172.21.15.136:6808/1223127568","heartbeat_front_addr":"172.21.15.136:6807/1223127568","state":["exists","up"]},{"osd":2,"uuid":"0e60acb1-d576-40cf-9b67-28f11a3a2846","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":1411885928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1411885928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1411885928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":1411885928}]},"public_addr":"172.21.15.136:6809/1411885928","cluster_addr":"172.21.15.136:6810/1411885928","heartbeat_back_addr":"172.21.15.136:6812/1411885928","heartbeat_front_addr":"172.21.15.136:6811/1411885928","state":["exists","up"]},{"osd":3,"uuid":"c52d96a6-a66d-446b-aa35-891cc84fa99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":802386900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":802386900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":802386900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6815","nonce":802386900}]},"public_addr":"172.21.15.136:6813/802386900","cluster_addr":"172.21.15.136:6814/802386900","heartbeat_back_addr":"172.21.15.136:6816/802386900","heartbeat_front_addr":"172.21.15.136:6815/802386900","state":["exists","up"]},{"osd":4,"uuid":"a304c9c4-c2c8-4e3d-8292-48bc13c48ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":952209544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":952209544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":952209544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":952209544}]},"public_addr":"172.21.15.164:6800/952209544","cluster_addr":"172.21.15.164:6801/952209544","heartbeat_back_addr":"172.21.15.164:6803/952209544","heartbeat_front_addr":"172.21.15.164:6802/952209544","state":["exists","up"]},{"osd":5,"uuid":"0dadc6cb-52c2-409a-b2ef-b6535d8ef50e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2714562868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":2714562868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2714562868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2714562868}]},"public_addr":"172.21.15.164:6804/2714562868","cluster_addr":"172.21.15.164:6805/2714562868","heartbeat_back_addr":"172.21.15.164:6807/2714562868","heartbeat_front_addr":"172.21.15.164:6806/2714562868","state":["exists","up"]},{"osd":6,"uuid":"018312be-ccea-4a85-9fa9-07154e0e0f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":268837773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6809","nonce":268837773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":268837773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":268837773}]},"public_addr":"172.21.15.164:6808/268837773","cluster_addr":"172.21.15.164:6809/268837773","heartbeat_back_addr":"172.21.15.164:6811/268837773","heartbeat_front_addr":"172.21.15.164:6810/268837773","state":["exists","up"]},{"osd":7,"uuid":"f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":683372681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":683372681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":683372681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":683372681}]},"public_addr":"172.21.15.164:6812/683372681","cluster_addr":"172.21.15.164:6813/683372681","heartbeat_back_addr":"172.21.15.164:6815/683372681","heartbeat_front_addr":"172.21.15.164:6814/683372681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:18:21.637001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:18:47.061150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:19:12.511638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:19:38.406167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:01.579742+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:25.928726+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:51.081171+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:21:15.322145+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/3403923406":"2024-03-27T05:16:56.499053+0000","172.21.15.136:0/3432366522":"2024-03-27T05:16:56.499053+0000","172.21.15.136:0/3430411216":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/3098548195":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/404149158":"2024-03-27T05:16:29.672020+0000","172.21.15.136:6800/1232474076":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/747850292":"2024-03-27T05:16:56.499053+0000","172.21.15.136:6800/3861450553":"2024-03-27T05:16:56.499053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4c5406da70 msgr2=0x7f4c5406ff30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 --2- 172.21.15.136:0/257942043 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4c5406da70 0x7f4c5406ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c1022c0 msgr2=0x7f4c6c197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c1022c0 0x7f4c6c197d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000ec10 tx=0x7f4c6000ef20 comp rx=0 tx=0).stop 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 shutdown_connections 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 --2- 172.21.15.136:0/257942043 >> v2:172.21.15.136:6800/2365933975 conn(0x7f4c5406da70 0x7f4c5406ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c6c1073a0 0x7f4c6c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c6c1022c0 0x7f4c6c197d90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 --2- 172.21.15.136:0/257942043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6c102c70 0x7f4c6c1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 >> 172.21.15.136:0/257942043 conn(0x7f4c6c0fdb30 msgr2=0x7f4c6c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 shutdown_connections 2024-03-26T05:22:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:07.764+0000 7f4c70b50700 1 -- 172.21.15.136:0/257942043 wait complete. 2024-03-26T05:22:07.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/257942043' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:22:07.968 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:07 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:07.829+0000 7f1084c1d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T05:22:08.019 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/257942043' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:22:08.019 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:07.752+0000 7f21cd298200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T05:22:08.019 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/257942043' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:22:08.323 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:08 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:08.000+0000 7f21cd298200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T05:22:08.694 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T05:22:08.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T05:22:08.948 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:08.967 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:08 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:08.625+0000 7f1084c1d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T05:22:09.067 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:08 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:08.824+0000 7f21cd298200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T05:22:09.467 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:09.197+0000 7f1084c1d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T05:22:09.467 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:09.295+0000 7f1084c1d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T05:22:09.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:09.442+0000 7f21cd298200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T05:22:09.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:09.549+0000 7f21cd298200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T05:22:09.752 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-mon[33554]: Standby manager daemon x restarted 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-mon[33554]: Standby manager daemon x started 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-mon[33554]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-mon[33554]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-mon[33554]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-mon[33554]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:09.513+0000 7f1084c1d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: [26/Mar/2024:05:22:09] ENGINE Bus STARTING 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: CherryPy Checker: 2024-03-26T05:22:09.753 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: The Application mounted at '' has an empty config. 2024-03-26T05:22:09.754 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: 2024-03-26T05:22:09.754 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: [26/Mar/2024:05:22:09] ENGINE Serving on http://:::9283 2024-03-26T05:22:09.754 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:22:09 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x[35438]: [26/Mar/2024:05:22:09] ENGINE Bus STARTED 2024-03-26T05:22:09.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[38106]: Standby manager daemon x restarted 2024-03-26T05:22:09.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[38106]: Standby manager daemon x started 2024-03-26T05:22:09.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[38106]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:22:09.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[38106]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:22:09.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[38106]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:22:09.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[38106]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:22:09.886 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:09.785+0000 7f21cd298200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T05:22:09.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[29560]: Standby manager daemon x restarted 2024-03-26T05:22:09.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[29560]: Standby manager daemon x started 2024-03-26T05:22:09.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[29560]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T05:22:09.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[29560]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T05:22:09.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[29560]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T05:22:09.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-mon[29560]: from='mgr.? 172.21.15.164:0/2126859464' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T05:22:10.310 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:09 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:09] ENGINE Bus STARTING 2024-03-26T05:22:10.310 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: CherryPy Checker: 2024-03-26T05:22:10.310 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: The Application mounted at '' has an empty config. 2024-03-26T05:22:10.311 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:22:10.311 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:10] ENGINE Serving on http://:::9283 2024-03-26T05:22:10.311 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:10] ENGINE Bus STARTED 2024-03-26T05:22:10.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: Active manager daemon y restarted 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: Activating manager daemon y 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: mgrmap e19: y(active, starting, since 0.0086868s), standbys: x 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:22:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: Manager daemon y is now available 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:22:10.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: Active manager daemon y restarted 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: Activating manager daemon y 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: mgrmap e19: y(active, starting, since 0.0086868s), standbys: x 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:22:10.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: Manager daemon y is now available 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:10.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:10.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:10.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:10.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:22:10.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:22:10.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:22:10.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:22:10.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:22:10.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:10 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:22:10.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.969+0000 7f5d2fc56700 1 -- 172.21.15.136:0/1823563469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28101bc0 msgr2=0x7f5d28105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:10.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.969+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/1823563469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28101bc0 0x7f5d28105bb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009990 tx=0x7f5d18009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:10.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.969+0000 7f5d2fc56700 1 -- 172.21.15.136:0/1823563469 shutdown_connections 2024-03-26T05:22:10.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.969+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/1823563469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d28106290 0x7f5d2810de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:10.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.969+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/1823563469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d28101210 0x7f5d281015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.969+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/1823563469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28101bc0 0x7f5d28105bb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.969+0000 7f5d2fc56700 1 -- 172.21.15.136:0/1823563469 >> 172.21.15.136:0/1823563469 conn(0x7f5d28076fc0 msgr2=0x7f5d280773d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 -- 172.21.15.136:0/1823563469 shutdown_connections 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 -- 172.21.15.136:0/1823563469 wait complete. 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 Processor -- start 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 -- start start 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d28101210 0x7f5d28198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d28101bc0 0x7f5d281990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28106290 0x7f5d281a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d28199980 con 0x7f5d28106290 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d281a4160 con 0x7f5d28101210 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.970+0000 7f5d2fc56700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d281a42a0 con 0x7f5d28101bc0 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2d1f1700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d28101bc0 0x7f5d281990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2e1f3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28106290 0x7f5d281a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:10.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2d1f1700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d28101bc0 0x7f5d281990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:47678/0 (socket says 172.21.15.136:47678) 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2d1f1700 1 -- 172.21.15.136:0/6883887 learned_addr learned my addr 172.21.15.136:0/6883887 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2d9f2700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d28101210 0x7f5d28198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2e1f3700 1 -- 172.21.15.136:0/6883887 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d28101bc0 msgr2=0x7f5d281990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2e1f3700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d28101bc0 0x7f5d281990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2e1f3700 1 -- 172.21.15.136:0/6883887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d28101210 msgr2=0x7f5d28198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2e1f3700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d28101210 0x7f5d28198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2e1f3700 1 -- 172.21.15.136:0/6883887 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d24009520 con 0x7f5d28106290 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2d1f1700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d28101bc0 0x7f5d281990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2e1f3700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28106290 0x7f5d281a3c20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009f60 tx=0x7f5d20009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d1effd700 1 -- 172.21.15.136:0/6883887 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d20018070 con 0x7f5d28106290 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d1effd700 1 -- 172.21.15.136:0/6883887 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5d2000f040 con 0x7f5d28106290 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d1effd700 1 -- 172.21.15.136:0/6883887 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2000cd10 con 0x7f5d28106290 2024-03-26T05:22:10.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d180095f0 con 0x7f5d28106290 2024-03-26T05:22:10.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.971+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d281a4810 con 0x7f5d28106290 2024-03-26T05:22:10.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.972+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d28068d40 con 0x7f5d28106290 2024-03-26T05:22:10.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.973+0000 7f5d1effd700 1 -- 172.21.15.136:0/6883887 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 20) v1 ==== 91172+0+0 (secure 0 0 0) 0x7f5d20003a00 con 0x7f5d28106290 2024-03-26T05:22:10.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.973+0000 7f5d1effd700 1 --2- 172.21.15.136:0/6883887 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5d1406d790 0x7f5d1406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:10.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.973+0000 7f5d1effd700 1 -- 172.21.15.136:0/6883887 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f5d2008c9c0 con 0x7f5d28106290 2024-03-26T05:22:10.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.973+0000 7f5d2d9f2700 1 --2- 172.21.15.136:0/6883887 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5d1406d790 0x7f5d1406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:10.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.974+0000 7f5d2d9f2700 1 --2- 172.21.15.136:0/6883887 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5d1406d790 0x7f5d1406fc50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009df0 tx=0x7f5d24009260 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:10.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: Active manager daemon y restarted 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: Activating manager daemon y 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: mgrmap e19: y(active, starting, since 0.0086868s), standbys: x 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T05:22:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: Manager daemon y is now available 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:22:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:10 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T05:22:10.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:10.978+0000 7f5d1effd700 1 -- 172.21.15.136:0/6883887 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5d20056560 con 0x7f5d28106290 2024-03-26T05:22:11.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.189+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d28110710 con 0x7f5d28106290 2024-03-26T05:22:11.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.191+0000 7f5d1effd700 1 -- 172.21.15.136:0/6883887 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+21073 (secure 0 0 0) 0x7f5d20059c20 con 0x7f5d28106290 2024-03-26T05:22:11.181 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:22:11.181 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":72,"fsid":"d69b39a8-eb2f-11ee-95d0-87774f69a715","created":"2024-03-26T05:15:55.712909+0000","modified":"2024-03-26T05:22:09.799823+0000","last_up_change":"2024-03-26T05:21:17.162399+0000","last_in_change":"2024-03-26T05:20:59.552150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T05:19:14.832191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T05:21:28.746312+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T05:21:29.507514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T05:21:31.156582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T05:21:33.087723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T05:21:35.140790+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d4698473-d228-4853-b404-54b255f0a9d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":1923051194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1923051194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1923051194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":1923051194}]},"public_addr":"172.21.15.136:6801/1923051194","cluster_addr":"172.21.15.136:6802/1923051194","heartbeat_back_addr":"172.21.15.136:6804/1923051194","heartbeat_front_addr":"172.21.15.136:6803/1923051194","state":["exists","up"]},{"osd":1,"uuid":"800f61d3-4725-45b6-af8c-0653cb8b4c82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":1223127568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1223127568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1223127568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":1223127568}]},"public_addr":"172.21.15.136:6805/1223127568","cluster_addr":"172.21.15.136:6806/1223127568","heartbeat_back_addr":"172.21.15.136:6808/1223127568","heartbeat_front_addr":"172.21.15.136:6807/1223127568","state":["exists","up"]},{"osd":2,"uuid":"0e60acb1-d576-40cf-9b67-28f11a3a2846","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":1411885928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1411885928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1411885928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":1411885928}]},"public_addr":"172.21.15.136:6809/1411885928","cluster_addr":"172.21.15.136:6810/1411885928","heartbeat_back_addr":"172.21.15.136:6812/1411885928","heartbeat_front_addr":"172.21.15.136:6811/1411885928","state":["exists","up"]},{"osd":3,"uuid":"c52d96a6-a66d-446b-aa35-891cc84fa99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":802386900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":802386900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":802386900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6815","nonce":802386900}]},"public_addr":"172.21.15.136:6813/802386900","cluster_addr":"172.21.15.136:6814/802386900","heartbeat_back_addr":"172.21.15.136:6816/802386900","heartbeat_front_addr":"172.21.15.136:6815/802386900","state":["exists","up"]},{"osd":4,"uuid":"a304c9c4-c2c8-4e3d-8292-48bc13c48ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":952209544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":952209544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":952209544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":952209544}]},"public_addr":"172.21.15.164:6800/952209544","cluster_addr":"172.21.15.164:6801/952209544","heartbeat_back_addr":"172.21.15.164:6803/952209544","heartbeat_front_addr":"172.21.15.164:6802/952209544","state":["exists","up"]},{"osd":5,"uuid":"0dadc6cb-52c2-409a-b2ef-b6535d8ef50e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2714562868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":2714562868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":2714562868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2714562868}]},"public_addr":"172.21.15.164:6804/2714562868","cluster_addr":"172.21.15.164:6805/2714562868","heartbeat_back_addr":"172.21.15.164:6807/2714562868","heartbeat_front_addr":"172.21.15.164:6806/2714562868","state":["exists","up"]},{"osd":6,"uuid":"018312be-ccea-4a85-9fa9-07154e0e0f15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":268837773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6809","nonce":268837773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":268837773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":268837773}]},"public_addr":"172.21.15.164:6808/268837773","cluster_addr":"172.21.15.164:6809/268837773","heartbeat_back_addr":"172.21.15.164:6811/268837773","heartbeat_front_addr":"172.21.15.164:6810/268837773","state":["exists","up"]},{"osd":7,"uuid":"f84c18e6-db08-4ecd-b6e3-9ba7bd0751bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":683372681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":683372681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":683372681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":683372681}]},"public_addr":"172.21.15.164:6812/683372681","cluster_addr":"172.21.15.164:6813/683372681","heartbeat_back_addr":"172.21.15.164:6815/683372681","heartbeat_front_addr":"172.21.15.164:6814/683372681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:18:21.637001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:18:47.061150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:19:12.511638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:19:38.406167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:01.579742+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:25.928726+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:20:51.081171+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T05:21:15.322145+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/1158073833":"2024-03-27T05:22:09.799773+0000","172.21.15.136:0/28828828":"2024-03-27T05:22:09.799773+0000","172.21.15.136:0/2667793025":"2024-03-27T05:22:09.799773+0000","172.21.15.136:0/3403923406":"2024-03-27T05:16:56.499053+0000","172.21.15.136:0/3432366522":"2024-03-27T05:16:56.499053+0000","172.21.15.136:6800/2365933975":"2024-03-27T05:22:09.799773+0000","172.21.15.136:0/3430411216":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/3098548195":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/404149158":"2024-03-27T05:16:29.672020+0000","172.21.15.136:6800/1232474076":"2024-03-27T05:16:29.672020+0000","172.21.15.136:0/1070436131":"2024-03-27T05:22:09.799773+0000","172.21.15.136:0/747850292":"2024-03-27T05:16:56.499053+0000","172.21.15.136:0/452904540":"2024-03-27T05:22:09.799773+0000","172.21.15.136:6800/3861450553":"2024-03-27T05:16:56.499053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T05:22:11.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5d1406d790 msgr2=0x7f5d1406fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:11.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/6883887 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5d1406d790 0x7f5d1406fc50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009df0 tx=0x7f5d24009260 comp rx=0 tx=0).stop 2024-03-26T05:22:11.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28106290 msgr2=0x7f5d281a3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:11.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28106290 0x7f5d281a3c20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009f60 tx=0x7f5d20009f90 comp rx=0 tx=0).stop 2024-03-26T05:22:11.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 shutdown_connections 2024-03-26T05:22:11.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/6883887 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5d1406d790 0x7f5d1406fc50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:11.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d28101210 0x7f5d28198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:11.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d28106290 0x7f5d281a3c20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:11.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 --2- 172.21.15.136:0/6883887 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d28101bc0 0x7f5d281990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:11.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.193+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 >> 172.21.15.136:0/6883887 conn(0x7f5d28076fc0 msgr2=0x7f5d280fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:11.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.194+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 shutdown_connections 2024-03-26T05:22:11.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:11.194+0000 7f5d2fc56700 1 -- 172.21.15.136:0/6883887 wait complete. 2024-03-26T05:22:11.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:11 smithi164 ceph-mon[33554]: [26/Mar/2024:05:22:10] ENGINE Bus STARTING 2024-03-26T05:22:11.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:11 smithi164 ceph-mon[33554]: [26/Mar/2024:05:22:10] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T05:22:11.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:11 smithi164 ceph-mon[33554]: mgrmap e20: y(active, since 1.01251s), standbys: x 2024-03-26T05:22:11.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/6883887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:22:12.029 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:11 smithi136 ceph-mon[29560]: [26/Mar/2024:05:22:10] ENGINE Bus STARTING 2024-03-26T05:22:12.029 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:11 smithi136 ceph-mon[29560]: [26/Mar/2024:05:22:10] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T05:22:12.030 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:11 smithi136 ceph-mon[29560]: mgrmap e20: y(active, since 1.01251s), standbys: x 2024-03-26T05:22:12.030 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/6883887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:22:12.030 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:11 smithi136 ceph-mon[38106]: [26/Mar/2024:05:22:10] ENGINE Bus STARTING 2024-03-26T05:22:12.030 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:11 smithi136 ceph-mon[38106]: [26/Mar/2024:05:22:10] ENGINE Serving on http://172.21.15.136:8765 2024-03-26T05:22:12.030 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:11 smithi136 ceph-mon[38106]: mgrmap e20: y(active, since 1.01251s), standbys: x 2024-03-26T05:22:12.030 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/6883887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T05:22:12.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T05:22:12.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T05:22:12.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T05:22:12.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T05:22:12.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T05:22:12.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T05:22:12.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T05:22:12.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T05:22:12.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:22:11 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:22:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:12.645 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:12.646 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:12.924 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:12 smithi136 ceph-mon[38106]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:12.924 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:12 smithi136 ceph-mon[38106]: [26/Mar/2024:05:22:10] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T05:22:12.924 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:12 smithi136 ceph-mon[38106]: [26/Mar/2024:05:22:10] ENGINE Bus STARTED 2024-03-26T05:22:12.924 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:12 smithi136 ceph-mon[29560]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:12.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:12 smithi136 ceph-mon[29560]: [26/Mar/2024:05:22:10] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T05:22:12.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:12 smithi136 ceph-mon[29560]: [26/Mar/2024:05:22:10] ENGINE Bus STARTED 2024-03-26T05:22:12.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:12 smithi164 ceph-mon[33554]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:12.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:12 smithi164 ceph-mon[33554]: [26/Mar/2024:05:22:10] ENGINE Serving on https://172.21.15.136:7150 2024-03-26T05:22:12.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:12 smithi164 ceph-mon[33554]: [26/Mar/2024:05:22:10] ENGINE Bus STARTED 2024-03-26T05:22:13.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[38106]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:13.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:13.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[38106]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T05:22:13.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:13.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:13.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[29560]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:13.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:13.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[29560]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T05:22:13.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:13.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:13 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:13.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:13 smithi164 ceph-mon[33554]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:13.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:13.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:13 smithi164 ceph-mon[33554]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T05:22:13.967 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:13 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:13 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi164 to 3760M 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[38106]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi164 to 3760M 2024-03-26T05:22:15.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:15 smithi136 ceph-mon[29560]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T05:22:15.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi164 to 3760M 2024-03-26T05:22:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:15 smithi164 ceph-mon[33554]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T05:22:16.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 -- 172.21.15.136:0/884946319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f253c06aa20 msgr2=0x7f253c06ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:16.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 --2- 172.21.15.136:0/884946319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f253c06aa20 0x7f253c06ae00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f252c0098f0 tx=0x7f252c009c00 comp rx=0 tx=0).stop 2024-03-26T05:22:16.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 -- 172.21.15.136:0/884946319 shutdown_connections 2024-03-26T05:22:16.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 --2- 172.21.15.136:0/884946319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f253c1074c0 0x7f253c10f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:16.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 --2- 172.21.15.136:0/884946319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f253c06aa20 0x7f253c06ae00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:16.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 --2- 172.21.15.136:0/884946319 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c06b3d0 0x7f253c106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:16.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 -- 172.21.15.136:0/884946319 >> 172.21.15.136:0/884946319 conn(0x7f253c077de0 msgr2=0x7f253c0781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:16.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 -- 172.21.15.136:0/884946319 shutdown_connections 2024-03-26T05:22:16.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 -- 172.21.15.136:0/884946319 wait complete. 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 Processor -- start 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.186+0000 7f2542419700 1 -- start start 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f2542419700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c06aa20 0x7f253c19da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f2542419700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f253c06b3d0 0x7f253c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f2542419700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f253c1074c0 0x7f253c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f2542419700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c1982d0 con 0x7f253c06b3d0 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f2542419700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c198440 con 0x7f253c06aa20 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f2542419700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c198600 con 0x7f253c1074c0 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f253c06b3d0 0x7f253c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f25409b6700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f253c1074c0 0x7f253c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:16.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f253c06b3d0 0x7f253c19df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40160/0 (socket says 172.21.15.136:40160) 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253b7fe700 1 -- 172.21.15.136:0/4208362991 learned_addr learned my addr 172.21.15.136:0/4208362991 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253bfff700 1 --2- 172.21.15.136:0/4208362991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c06aa20 0x7f253c19da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253b7fe700 1 -- 172.21.15.136:0/4208362991 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f253c1074c0 msgr2=0x7f253c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253b7fe700 1 --2- 172.21.15.136:0/4208362991 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f253c1074c0 0x7f253c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253b7fe700 1 -- 172.21.15.136:0/4208362991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c06aa20 msgr2=0x7f253c19da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253b7fe700 1 --2- 172.21.15.136:0/4208362991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c06aa20 0x7f253c19da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.187+0000 7f253b7fe700 1 -- 172.21.15.136:0/4208362991 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2534009d00 con 0x7f253c06b3d0 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.188+0000 7f253b7fe700 1 --2- 172.21.15.136:0/4208362991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f253c06b3d0 0x7f253c19df90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f253000d980 tx=0x7f253000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.188+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530009930 con 0x7f253c06b3d0 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.188+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f2530010460 con 0x7f253c06b3d0 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.188+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c009520 con 0x7f253c06b3d0 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.188+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c1a49d0 con 0x7f253c06b3d0 2024-03-26T05:22:16.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.188+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000f620 con 0x7f253c06b3d0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.189+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2528000f80 con 0x7f253c06b3d0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.190+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f253000f780 con 0x7f253c06b3d0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.190+0000 7f25397fa700 1 --2- 172.21.15.136:0/4208362991 >> v2:172.21.15.136:6800/3380174081 conn(0x7f252406da80 0x7f252406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.190+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f253008cd90 con 0x7f253c06b3d0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.190+0000 7f253bfff700 1 --2- 172.21.15.136:0/4208362991 >> v2:172.21.15.136:6800/3380174081 conn(0x7f252406da80 0x7f252406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.190+0000 7f25397fa700 1 --2- 172.21.15.136:0/4208362991 >> v2:172.21.15.164:6808/268837773 conn(0x7f25240773a0 0x7f2524079840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.190+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 --> v2:172.21.15.164:6808/268837773 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2524079ed0 con 0x7f25240773a0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.190+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f253008f450 con 0x7f253c06b3d0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.191+0000 7f253bfff700 1 --2- 172.21.15.136:0/4208362991 >> v2:172.21.15.136:6800/3380174081 conn(0x7f252406da80 0x7f252406ff40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f252c009fd0 tx=0x7f252c019040 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.191+0000 7f25409b6700 1 --2- 172.21.15.136:0/4208362991 >> v2:172.21.15.164:6808/268837773 conn(0x7f25240773a0 0x7f2524079840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:16.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.191+0000 7f25409b6700 1 --2- 172.21.15.136:0/4208362991 >> v2:172.21.15.164:6808/268837773 conn(0x7f25240773a0 0x7f2524079840 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:16.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.192+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 <== osd.6 v2:172.21.15.164:6808/268837773 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2524079ed0 con 0x7f25240773a0 2024-03-26T05:22:16.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.208+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 --> v2:172.21.15.164:6808/268837773 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2528002c60 con 0x7f25240773a0 2024-03-26T05:22:16.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f25397fa700 1 -- 172.21.15.136:0/4208362991 <== osd.6 v2:172.21.15.164:6808/268837773 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2528002c60 con 0x7f25240773a0 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 >> v2:172.21.15.164:6808/268837773 conn(0x7f25240773a0 msgr2=0x7f2524079840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 --2- 172.21.15.136:0/4208362991 >> v2:172.21.15.164:6808/268837773 conn(0x7f25240773a0 0x7f2524079840 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 >> v2:172.21.15.136:6800/3380174081 conn(0x7f252406da80 msgr2=0x7f252406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 --2- 172.21.15.136:0/4208362991 >> v2:172.21.15.136:6800/3380174081 conn(0x7f252406da80 0x7f252406ff40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f252c009fd0 tx=0x7f252c019040 comp rx=0 tx=0).stop 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f253c06b3d0 msgr2=0x7f253c19df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 --2- 172.21.15.136:0/4208362991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f253c06b3d0 0x7f253c19df90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f253000d980 tx=0x7f253000dc90 comp rx=0 tx=0).stop 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f25409b6700 1 -- 172.21.15.136:0/4208362991 reap_dead start 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 shutdown_connections 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 >> 172.21.15.136:0/4208362991 conn(0x7f253c077de0 msgr2=0x7f253c0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 shutdown_connections 2024-03-26T05:22:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:16.209+0000 7f2542419700 1 -- 172.21.15.136:0/4208362991 wait complete. 2024-03-26T05:22:17.140 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:17 smithi136 ceph-mon[38106]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T05:22:17.141 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:17 smithi136 ceph-mon[29560]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T05:22:17.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:17 smithi164 ceph-mon[33554]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T05:22:17.559 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:22:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 -- 172.21.15.136:0/2058325771 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884106290 msgr2=0x7f188410de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 --2- 172.21.15.136:0/2058325771 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884106290 0x7f188410de60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f187c00b0a0 tx=0x7f187c00b3b0 comp rx=0 tx=0).stop 2024-03-26T05:22:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 -- 172.21.15.136:0/2058325771 shutdown_connections 2024-03-26T05:22:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 --2- 172.21.15.136:0/2058325771 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884106290 0x7f188410de60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 --2- 172.21.15.136:0/2058325771 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1884101210 0x7f18841015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 --2- 172.21.15.136:0/2058325771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1884101bc0 0x7f1884105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 -- 172.21.15.136:0/2058325771 >> 172.21.15.136:0/2058325771 conn(0x7f1884076f90 msgr2=0x7f18840773a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 -- 172.21.15.136:0/2058325771 shutdown_connections 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.969+0000 7f1889b1f700 1 -- 172.21.15.136:0/2058325771 wait complete. 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.970+0000 7f1889b1f700 1 Processor -- start 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.970+0000 7f1889b1f700 1 -- start start 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.970+0000 7f1889b1f700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884101210 0x7f1884198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.970+0000 7f1889b1f700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1884101bc0 0x7f1884199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.970+0000 7f1889b1f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1884106290 0x7f18841a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.970+0000 7f1889b1f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1884199a50 con 0x7f1884106290 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.970+0000 7f1889b1f700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18841a4210 con 0x7f1884101bc0 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.970+0000 7f1889b1f700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18841a4350 con 0x7f1884101210 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884101210 0x7f1884198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f1883fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1884106290 0x7f18841a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884101210 0x7f1884198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:47738/0 (socket says 172.21.15.136:47738) 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 -- 172.21.15.136:0/4010607251 learned_addr learned my addr 172.21.15.136:0/4010607251 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f1882ffd700 1 --2- 172.21.15.136:0/4010607251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1884101bc0 0x7f1884199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 -- 172.21.15.136:0/4010607251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1884101bc0 msgr2=0x7f1884199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 --2- 172.21.15.136:0/4010607251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1884101bc0 0x7f1884199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 -- 172.21.15.136:0/4010607251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1884106290 msgr2=0x7f18841a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 --2- 172.21.15.136:0/4010607251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1884106290 0x7f18841a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 -- 172.21.15.136:0/4010607251 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874009520 con 0x7f1884101210 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f1883fff700 1 --2- 172.21.15.136:0/4010607251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1884106290 0x7f18841a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f18837fe700 1 --2- 172.21.15.136:0/4010607251 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884101210 0x7f1884198c50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f187400eb30 tx=0x7f187400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874009880 con 0x7f1884101210 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f18740044d0 con 0x7f1884101210 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187c009d00 con 0x7f1884101210 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874003e20 con 0x7f1884101210 2024-03-26T05:22:17.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.971+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18841a49a0 con 0x7f1884101210 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.972+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1884110a40 con 0x7f1884101210 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.973+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f1874003f80 con 0x7f1884101210 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.973+0000 7f1880ff9700 1 --2- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6800/3380174081 conn(0x7f186c06da80 0x7f186c06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.973+0000 7f1882ffd700 1 --2- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6800/3380174081 conn(0x7f186c06da80 0x7f186c06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.973+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f1874014070 con 0x7f1884101210 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.973+0000 7f1880ff9700 1 --2- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6809/1411885928 conn(0x7f186c077320 0x7f186c0797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.973+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 --> v2:172.21.15.136:6809/1411885928 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f186c079e50 con 0x7f186c077320 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.973+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f187408dd00 con 0x7f1884101210 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.973+0000 7f1883fff700 1 --2- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6809/1411885928 conn(0x7f186c077320 0x7f186c0797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.974+0000 7f1882ffd700 1 --2- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6800/3380174081 conn(0x7f186c06da80 0x7f186c06ff40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1878005fd0 tx=0x7f1878005ee0 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.974+0000 7f1883fff700 1 --2- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6809/1411885928 conn(0x7f186c077320 0x7f186c0797c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:17.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.974+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 <== osd.2 v2:172.21.15.136:6809/1411885928 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f186c079e50 con 0x7f186c077320 2024-03-26T05:22:17.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 --> v2:172.21.15.136:6809/1411885928 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1884062710 con 0x7f186c077320 2024-03-26T05:22:17.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1880ff9700 1 -- 172.21.15.136:0/4010607251 <== osd.2 v2:172.21.15.136:6809/1411885928 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1884062710 con 0x7f186c077320 2024-03-26T05:22:17.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6809/1411885928 conn(0x7f186c077320 msgr2=0x7f186c0797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1889b1f700 1 --2- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6809/1411885928 conn(0x7f186c077320 0x7f186c0797c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6800/3380174081 conn(0x7f186c06da80 msgr2=0x7f186c06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1889b1f700 1 --2- 172.21.15.136:0/4010607251 >> v2:172.21.15.136:6800/3380174081 conn(0x7f186c06da80 0x7f186c06ff40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1878005fd0 tx=0x7f1878005ee0 comp rx=0 tx=0).stop 2024-03-26T05:22:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884101210 msgr2=0x7f1884198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1889b1f700 1 --2- 172.21.15.136:0/4010607251 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1884101210 0x7f1884198c50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f187400eb30 tx=0x7f187400ee40 comp rx=0 tx=0).stop 2024-03-26T05:22:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1883fff700 1 -- 172.21.15.136:0/4010607251 reap_dead start 2024-03-26T05:22:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.991+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 shutdown_connections 2024-03-26T05:22:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.992+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 >> 172.21.15.136:0/4010607251 conn(0x7f1884076f90 msgr2=0x7f1884100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:17.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.992+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 shutdown_connections 2024-03-26T05:22:17.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:17.992+0000 7f1889b1f700 1 -- 172.21.15.136:0/4010607251 wait complete. 2024-03-26T05:22:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:19 smithi136 ceph-mon[38106]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T05:22:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:19 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:19 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:19.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:19 smithi136 ceph-mon[29560]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T05:22:19.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:19 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:19.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:19 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:19.372 INFO:teuthology.orchestra.run.smithi136.stdout:77309411367 2024-03-26T05:22:19.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T05:22:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:19 smithi164 ceph-mon[33554]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T05:22:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:19 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:19.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:19 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:19.616 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:19.618 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:19.618 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:20.083 INFO:teuthology.orchestra.run.smithi136.stdout:197568495634 2024-03-26T05:22:20.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 -- 172.21.15.136:0/4155624689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2541070f0 msgr2=0x7fb2541074d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 --2- 172.21.15.136:0/4155624689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2541070f0 0x7fb2541074d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb244009990 tx=0x7fb244009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 -- 172.21.15.136:0/4155624689 shutdown_connections 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 --2- 172.21.15.136:0/4155624689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb254101030 0x7fb25410ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 --2- 172.21.15.136:0/4155624689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2541070f0 0x7fb2541074d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 --2- 172.21.15.136:0/4155624689 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb254100670 0x7fb254100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 -- 172.21.15.136:0/4155624689 >> 172.21.15.136:0/4155624689 conn(0x7fb254077580 msgr2=0x7fb2540fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 -- 172.21.15.136:0/4155624689 shutdown_connections 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.292+0000 7fb2589da700 1 -- 172.21.15.136:0/4155624689 wait complete. 2024-03-26T05:22:20.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb2589da700 1 Processor -- start 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb2589da700 1 -- start start 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb2589da700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb254100670 0x7fb254101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb2589da700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb254101030 0x7fb254101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb2589da700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2541070f0 0x7fb2541022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb2589da700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2541a61a0 con 0x7fb2541070f0 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb2589da700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2541a6310 con 0x7fb254101030 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb2589da700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2541a64d0 con 0x7fb254100670 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb25259c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb254100670 0x7fb254101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb25259c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb254100670 0x7fb254101850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:47742/0 (socket says 172.21.15.136:47742) 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb25259c700 1 -- 172.21.15.136:0/3330670314 learned_addr learned my addr 172.21.15.136:0/3330670314 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb251d9b700 1 --2- 172.21.15.136:0/3330670314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb254101030 0x7fb254101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb25259c700 1 -- 172.21.15.136:0/3330670314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb254101030 msgr2=0x7fb254101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb25259c700 1 --2- 172.21.15.136:0/3330670314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb254101030 0x7fb254101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb25259c700 1 -- 172.21.15.136:0/3330670314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2541070f0 msgr2=0x7fb2541022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb25259c700 1 --2- 172.21.15.136:0/3330670314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2541070f0 0x7fb2541022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.294+0000 7fb25259c700 1 -- 172.21.15.136:0/3330670314 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248009520 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.295+0000 7fb25259c700 1 --2- 172.21.15.136:0/3330670314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb254100670 0x7fb254101850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb244005910 tx=0x7fb244004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.295+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24401d070 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.295+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fb244022470 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.295+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244017650 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.295+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2440095f0 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.295+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2541a6a30 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.296+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb254111c00 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.296+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7fb24400fb10 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.297+0000 7fb2437fe700 1 --2- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb23c06da80 0x7fb23c06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.297+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fb24408e680 con 0x7fb254100670 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.297+0000 7fb251d9b700 1 --2- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb23c06da80 0x7fb23c06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.297+0000 7fb2437fe700 1 --2- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6801/1923051194 conn(0x7fb23c0773a0 0x7fb23c079840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.297+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 --> v2:172.21.15.136:6801/1923051194 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb23c079ed0 con 0x7fb23c0773a0 2024-03-26T05:22:20.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.297+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb244090d10 con 0x7fb254100670 2024-03-26T05:22:20.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.297+0000 7fb252d9d700 1 --2- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6801/1923051194 conn(0x7fb23c0773a0 0x7fb23c079840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.297+0000 7fb251d9b700 1 --2- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb23c06da80 0x7fb23c06ff40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2541122b0 tx=0x7fb248009260 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:20.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.298+0000 7fb252d9d700 1 --2- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6801/1923051194 conn(0x7fb23c0773a0 0x7fb23c079840 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:20.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.298+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 <== osd.0 v2:172.21.15.136:6801/1923051194 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb23c079ed0 con 0x7fb23c0773a0 2024-03-26T05:22:20.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.318+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 --> v2:172.21.15.136:6801/1923051194 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb254105510 con 0x7fb23c0773a0 2024-03-26T05:22:20.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.318+0000 7fb2437fe700 1 -- 172.21.15.136:0/3330670314 <== osd.0 v2:172.21.15.136:6801/1923051194 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb254105510 con 0x7fb23c0773a0 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.318+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6801/1923051194 conn(0x7fb23c0773a0 msgr2=0x7fb23c079840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.318+0000 7fb2589da700 1 --2- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6801/1923051194 conn(0x7fb23c0773a0 0x7fb23c079840 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb23c06da80 msgr2=0x7fb23c06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb2589da700 1 --2- 172.21.15.136:0/3330670314 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb23c06da80 0x7fb23c06ff40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2541122b0 tx=0x7fb248009260 comp rx=0 tx=0).stop 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb254100670 msgr2=0x7fb254101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb2589da700 1 --2- 172.21.15.136:0/3330670314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb254100670 0x7fb254101850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb244005910 tx=0x7fb244004730 comp rx=0 tx=0).stop 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb252d9d700 1 -- 172.21.15.136:0/3330670314 reap_dead start 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 shutdown_connections 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 >> 172.21.15.136:0/3330670314 conn(0x7fb254077580 msgr2=0x7fb254104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:20.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 shutdown_connections 2024-03-26T05:22:20.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.319+0000 7fb2589da700 1 -- 172.21.15.136:0/3330670314 wait complete. 2024-03-26T05:22:20.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 -- 172.21.15.136:0/1570303690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c1019e0 msgr2=0x7ff52c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 --2- 172.21.15.136:0/1570303690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c1019e0 0x7ff52c101dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0098f0 tx=0x7ff51c009c00 comp rx=0 tx=0).stop 2024-03-26T05:22:20.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 -- 172.21.15.136:0/1570303690 shutdown_connections 2024-03-26T05:22:20.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 --2- 172.21.15.136:0/1570303690 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52c106300 0x7ff52c10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 --2- 172.21.15.136:0/1570303690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c1019e0 0x7ff52c101dc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 --2- 172.21.15.136:0/1570303690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52c102390 0x7ff52c105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 -- 172.21.15.136:0/1570303690 >> 172.21.15.136:0/1570303690 conn(0x7ff52c077580 msgr2=0x7ff52c0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:20.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 -- 172.21.15.136:0/1570303690 shutdown_connections 2024-03-26T05:22:20.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 -- 172.21.15.136:0/1570303690 wait complete. 2024-03-26T05:22:20.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 Processor -- start 2024-03-26T05:22:20.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.819+0000 7ff532133700 1 -- start start 2024-03-26T05:22:20.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff532133700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52c1019e0 0x7ff52c0fca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff532133700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52c102390 0x7ff52c0fcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff532133700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c106300 0x7ff52c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff532133700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1a6180 con 0x7ff52c106300 2024-03-26T05:22:20.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff532133700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1a62f0 con 0x7ff52c102390 2024-03-26T05:22:20.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff532133700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1a64b0 con 0x7ff52c1019e0 2024-03-26T05:22:20.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c106300 0x7ff52c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52b7fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52c1019e0 0x7ff52c0fca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52affd700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52c102390 0x7ff52c0fcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c106300 0x7ff52c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:48610/0 (socket says 172.21.15.136:48610) 2024-03-26T05:22:20.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 -- 172.21.15.136:0/1618989867 learned_addr learned my addr 172.21.15.136:0/1618989867 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:20.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 -- 172.21.15.136:0/1618989867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52c1019e0 msgr2=0x7ff52c0fca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 --2- 172.21.15.136:0/1618989867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52c1019e0 0x7ff52c0fca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 -- 172.21.15.136:0/1618989867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52c102390 msgr2=0x7ff52c0fcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 --2- 172.21.15.136:0/1618989867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52c102390 0x7ff52c0fcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 -- 172.21.15.136:0/1618989867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524009d00 con 0x7ff52c106300 2024-03-26T05:22:20.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52affd700 1 --2- 172.21.15.136:0/1618989867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52c102390 0x7ff52c0fcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:20.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff52bfff700 1 --2- 172.21.15.136:0/1618989867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c106300 0x7ff52c0fd4e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e3c0 tx=0x7ff52400e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:20.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.820+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5240090d0 con 0x7ff52c106300 2024-03-26T05:22:20.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.821+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c009520 con 0x7ff52c106300 2024-03-26T05:22:20.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.821+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ff52400f040 con 0x7ff52c106300 2024-03-26T05:22:20.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.821+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524007bd0 con 0x7ff52c106300 2024-03-26T05:22:20.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.821+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c1a6b00 con 0x7ff52c106300 2024-03-26T05:22:20.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.822+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff52c110a90 con 0x7ff52c106300 2024-03-26T05:22:20.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.822+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7ff524007d30 con 0x7ff52c106300 2024-03-26T05:22:20.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.822+0000 7ff528ff9700 1 --2- 172.21.15.136:0/1618989867 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff51406da80 0x7ff51406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.823+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7ff52408d390 con 0x7ff52c106300 2024-03-26T05:22:20.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.823+0000 7ff52b7fe700 1 --2- 172.21.15.136:0/1618989867 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff51406da80 0x7ff51406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.823+0000 7ff528ff9700 1 --2- 172.21.15.136:0/1618989867 >> v2:172.21.15.164:6812/683372681 conn(0x7ff514077320 0x7ff5140797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:20.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.823+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 --> v2:172.21.15.164:6812/683372681 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff514079e50 con 0x7ff514077320 2024-03-26T05:22:20.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.823+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff52408fa50 con 0x7ff52c106300 2024-03-26T05:22:20.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.823+0000 7ff52affd700 1 --2- 172.21.15.136:0/1618989867 >> v2:172.21.15.164:6812/683372681 conn(0x7ff514077320 0x7ff5140797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:20.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.823+0000 7ff52b7fe700 1 --2- 172.21.15.136:0/1618989867 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff51406da80 0x7ff51406ff40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009fd0 tx=0x7ff51c019040 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:20.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.823+0000 7ff52affd700 1 --2- 172.21.15.136:0/1618989867 >> v2:172.21.15.164:6812/683372681 conn(0x7ff514077320 0x7ff5140797c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:20.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.824+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 <== osd.7 v2:172.21.15.164:6812/683372681 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff514079e50 con 0x7ff514077320 2024-03-26T05:22:20.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.840+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 --> v2:172.21.15.164:6812/683372681 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff52c1054e0 con 0x7ff514077320 2024-03-26T05:22:20.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff528ff9700 1 -- 172.21.15.136:0/1618989867 <== osd.7 v2:172.21.15.164:6812/683372681 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff52c1054e0 con 0x7ff514077320 2024-03-26T05:22:20.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 >> v2:172.21.15.164:6812/683372681 conn(0x7ff514077320 msgr2=0x7ff5140797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 --2- 172.21.15.136:0/1618989867 >> v2:172.21.15.164:6812/683372681 conn(0x7ff514077320 0x7ff5140797c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff51406da80 msgr2=0x7ff51406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 --2- 172.21.15.136:0/1618989867 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff51406da80 0x7ff51406ff40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009fd0 tx=0x7ff51c019040 comp rx=0 tx=0).stop 2024-03-26T05:22:20.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c106300 msgr2=0x7ff52c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:20.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 --2- 172.21.15.136:0/1618989867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52c106300 0x7ff52c0fd4e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e3c0 tx=0x7ff52400e6d0 comp rx=0 tx=0).stop 2024-03-26T05:22:20.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff52bfff700 1 -- 172.21.15.136:0/1618989867 reap_dead start 2024-03-26T05:22:20.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 shutdown_connections 2024-03-26T05:22:20.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 >> 172.21.15.136:0/1618989867 conn(0x7ff52c077580 msgr2=0x7ff52c1008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:20.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 shutdown_connections 2024-03-26T05:22:20.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:20.841+0000 7ff532133700 1 -- 172.21.15.136:0/1618989867 wait complete. 2024-03-26T05:22:21.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi136 to 3760M 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[38106]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.373 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi136 to 3760M 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:22:21.374 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:22:21.375 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:22:21.375 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:21 smithi136 ceph-mon[29560]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi136 to 3760M 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: Updating smithi136:/etc/ceph/ceph.conf 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:22:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-mon[33554]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T05:22:22.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:22:21 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:22:22.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[38106]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:22:22.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[38106]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:22:22.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[38106]: Deploying daemon alertmanager.a on smithi136 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[29560]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[29560]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:22 smithi136 ceph-mon[29560]: Deploying daemon alertmanager.a on smithi136 2024-03-26T05:22:22.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:22 smithi164 ceph-mon[33554]: Updating smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:22:22.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:22 smithi164 ceph-mon[33554]: Updating smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T05:22:22.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:22 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:22 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:22 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:22 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:22 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:22.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:22 smithi164 ceph-mon[33554]: Deploying daemon alertmanager.a on smithi136 2024-03-26T05:22:22.659 INFO:teuthology.orchestra.run.smithi136.stdout:34359738417 2024-03-26T05:22:22.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T05:22:23.105 INFO:teuthology.orchestra.run.smithi136.stdout:231928233999 2024-03-26T05:22:23.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T05:22:23.332 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:23.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:23 smithi136 ceph-mon[29560]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T05:22:23.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:23.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:23 smithi136 ceph-mon[38106]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T05:22:23.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:23.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:23 smithi164 ceph-mon[33554]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T05:22:23.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:24.189 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:25.322 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:25.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:25 smithi136 ceph-mon[38106]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T05:22:25.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:25 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:25.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:25 smithi136 ceph-mon[29560]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T05:22:25.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:25 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:25.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:25 smithi164 ceph-mon[33554]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T05:22:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:25 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 -- 172.21.15.136:0/913116886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcffc1020d0 msgr2=0x7fcffc102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 --2- 172.21.15.136:0/913116886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcffc1020d0 0x7fcffc102550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80099c0 tx=0x7fcfe8009cd0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 -- 172.21.15.136:0/913116886 shutdown_connections 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 --2- 172.21.15.136:0/913116886 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcffc1080b0 0x7fcffc108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 --2- 172.21.15.136:0/913116886 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcffc102a90 0x7fcffc10eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 --2- 172.21.15.136:0/913116886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcffc1020d0 0x7fcffc102550 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 -- 172.21.15.136:0/913116886 >> 172.21.15.136:0/913116886 conn(0x7fcffc0791b0 msgr2=0x7fcffc0795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 -- 172.21.15.136:0/913116886 shutdown_connections 2024-03-26T05:22:26.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 -- 172.21.15.136:0/913116886 wait complete. 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 Processor -- start 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 -- start start 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.429+0000 7fd002dc5700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcffc1020d0 0x7fcffc19d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd002dc5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcffc102a90 0x7fcffc19dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd002dc5700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcffc1080b0 0x7fcffc197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd002dc5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1980a0 con 0x7fcffc102a90 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd002dc5700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1981e0 con 0x7fcffc1080b0 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd002dc5700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1983a0 con 0x7fcffc1020d0 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcffc1020d0 0x7fcffc19d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcffc1020d0 0x7fcffc19d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:44174/0 (socket says 172.21.15.136:44174) 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 -- 172.21.15.136:0/722724427 learned_addr learned my addr 172.21.15.136:0/722724427 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fcffbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcffc102a90 0x7fcffc19dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:26.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd001362700 1 --2- 172.21.15.136:0/722724427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcffc1080b0 0x7fcffc197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:26.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 -- 172.21.15.136:0/722724427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcffc1080b0 msgr2=0x7fcffc197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:26.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 --2- 172.21.15.136:0/722724427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcffc1080b0 0x7fcffc197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 -- 172.21.15.136:0/722724427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcffc102a90 msgr2=0x7fcffc19dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:26.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 --2- 172.21.15.136:0/722724427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcffc102a90 0x7fcffc19dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 -- 172.21.15.136:0/722724427 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff4009c40 con 0x7fcffc1020d0 2024-03-26T05:22:26.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fcffbfff700 1 --2- 172.21.15.136:0/722724427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcffc102a90 0x7fcffc19dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:26.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd000b61700 1 --2- 172.21.15.136:0/722724427 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcffc1020d0 0x7fcffc19d830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcff000c9b0 tx=0x7fcff000ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:26.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0007b00 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe80095f0 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fcff0007c60 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0018660 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.430+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc1a4700 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.431+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcffc111ac0 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.432+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7fcff000f450 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.432+0000 7fcff9ffb700 1 --2- 172.21.15.136:0/722724427 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfec06da80 0x7fcfec06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.432+0000 7fcffbfff700 1 --2- 172.21.15.136:0/722724427 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfec06da80 0x7fcfec06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.432+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fcff008d380 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.432+0000 7fcff9ffb700 1 --2- 172.21.15.136:0/722724427 >> v2:172.21.15.164:6804/2714562868 conn(0x7fcfec077320 0x7fcfec0797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.432+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 --> v2:172.21.15.164:6804/2714562868 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfec079e50 con 0x7fcfec077320 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.432+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcff008fa40 con 0x7fcffc1020d0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.433+0000 7fd001362700 1 --2- 172.21.15.136:0/722724427 >> v2:172.21.15.164:6804/2714562868 conn(0x7fcfec077320 0x7fcfec0797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.433+0000 7fcffbfff700 1 --2- 172.21.15.136:0/722724427 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfec06da80 0x7fcfec06ff40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8000c00 tx=0x7fcfe8005fb0 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:26.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.433+0000 7fd001362700 1 --2- 172.21.15.136:0/722724427 >> v2:172.21.15.164:6804/2714562868 conn(0x7fcfec077320 0x7fcfec0797c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.434+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 <== osd.5 v2:172.21.15.164:6804/2714562868 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcfec079e50 con 0x7fcfec077320 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.450+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 --> v2:172.21.15.164:6804/2714562868 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcffc1075d0 con 0x7fcfec077320 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fcff9ffb700 1 -- 172.21.15.136:0/722724427 <== osd.5 v2:172.21.15.164:6804/2714562868 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcffc1075d0 con 0x7fcfec077320 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 >> v2:172.21.15.164:6804/2714562868 conn(0x7fcfec077320 msgr2=0x7fcfec0797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 --2- 172.21.15.136:0/722724427 >> v2:172.21.15.164:6804/2714562868 conn(0x7fcfec077320 0x7fcfec0797c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfec06da80 msgr2=0x7fcfec06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 --2- 172.21.15.136:0/722724427 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfec06da80 0x7fcfec06ff40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8000c00 tx=0x7fcfe8005fb0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcffc1020d0 msgr2=0x7fcffc19d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 --2- 172.21.15.136:0/722724427 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcffc1020d0 0x7fcffc19d830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcff000c9b0 tx=0x7fcff000ccc0 comp rx=0 tx=0).stop 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd001362700 1 -- 172.21.15.136:0/722724427 reap_dead start 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 shutdown_connections 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 >> 172.21.15.136:0/722724427 conn(0x7fcffc0791b0 msgr2=0x7fcffc105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 shutdown_connections 2024-03-26T05:22:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:26.451+0000 7fd002dc5700 1 -- 172.21.15.136:0/722724427 wait complete. 2024-03-26T05:22:27.451 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:27 smithi136 ceph-mon[38106]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T05:22:27.451 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:22:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:27.452 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:27 smithi136 ceph-mon[29560]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T05:22:27.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:27 smithi164 ceph-mon[33554]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.414+0000 7fa0f37b3700 1 -- 172.21.15.136:0/694640516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 msgr2=0x7fa0ec1073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.414+0000 7fa0f37b3700 1 --2- 172.21.15.136:0/694640516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 0x7fa0ec1073a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009990 tx=0x7fa0e8009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.414+0000 7fa0f37b3700 1 -- 172.21.15.136:0/694640516 shutdown_connections 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.414+0000 7fa0f37b3700 1 --2- 172.21.15.136:0/694640516 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ec101920 0x7fa0ec10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.414+0000 7fa0f37b3700 1 --2- 172.21.15.136:0/694640516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 0x7fa0ec1073a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.414+0000 7fa0f37b3700 1 --2- 172.21.15.136:0/694640516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec100f80 0x7fa0ec1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.414+0000 7fa0f37b3700 1 -- 172.21.15.136:0/694640516 >> 172.21.15.136:0/694640516 conn(0x7fa0ec077eb0 msgr2=0x7fa0ec0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.415+0000 7fa0f37b3700 1 -- 172.21.15.136:0/694640516 shutdown_connections 2024-03-26T05:22:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.415+0000 7fa0f37b3700 1 -- 172.21.15.136:0/694640516 wait complete. 2024-03-26T05:22:28.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.415+0000 7fa0f37b3700 1 Processor -- start 2024-03-26T05:22:28.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.415+0000 7fa0f37b3700 1 -- start start 2024-03-26T05:22:28.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.415+0000 7fa0f37b3700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ec100f80 0x7fa0ec102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:28.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.415+0000 7fa0f37b3700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec101920 0x7fa0ec103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:28.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.415+0000 7fa0f37b3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 0x7fa0ec1036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:28.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f1d50700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 0x7fa0ec1036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:28.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f154f700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ec100f80 0x7fa0ec102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:28.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f37b3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec1a6030 con 0x7fa0ec106fc0 2024-03-26T05:22:28.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f1d50700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 0x7fa0ec1036b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:48640/0 (socket says 172.21.15.136:48640) 2024-03-26T05:22:28.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f1d50700 1 -- 172.21.15.136:0/3511325115 learned_addr learned my addr 172.21.15.136:0/3511325115 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:28.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec1a61a0 con 0x7fa0ec101920 2024-03-26T05:22:28.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec1a6360 con 0x7fa0ec100f80 2024-03-26T05:22:28.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f0d4e700 1 --2- 172.21.15.136:0/3511325115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec101920 0x7fa0ec103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:28.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f154f700 1 -- 172.21.15.136:0/3511325115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec101920 msgr2=0x7fa0ec103170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:28.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f154f700 1 --2- 172.21.15.136:0/3511325115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec101920 0x7fa0ec103170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f154f700 1 -- 172.21.15.136:0/3511325115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 msgr2=0x7fa0ec1036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f154f700 1 --2- 172.21.15.136:0/3511325115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 0x7fa0ec1036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f154f700 1 -- 172.21.15.136:0/3511325115 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e4009c40 con 0x7fa0ec100f80 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f1d50700 1 --2- 172.21.15.136:0/3511325115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ec106fc0 0x7fa0ec1036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f154f700 1 --2- 172.21.15.136:0/3511325115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ec100f80 0x7fa0ec102c30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80059d0 tx=0x7fa0e8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e801d070 con 0x7fa0ec100f80 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.416+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e80095f0 con 0x7fa0ec100f80 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.417+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa0e800b680 con 0x7fa0ec100f80 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.417+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8022e40 con 0x7fa0ec100f80 2024-03-26T05:22:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.417+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ec1a68c0 con 0x7fa0ec100f80 2024-03-26T05:22:28.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.417+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0ec1109c0 con 0x7fa0ec100f80 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.418+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7fa0e8022570 con 0x7fa0ec100f80 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.418+0000 7fa0e27fc700 1 --2- 172.21.15.136:0/3511325115 >> v2:172.21.15.136:6800/3380174081 conn(0x7fa0d806da80 0x7fa0d806ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.419+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fa0e808f1a0 con 0x7fa0ec100f80 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.419+0000 7fa0f0d4e700 1 --2- 172.21.15.136:0/3511325115 >> v2:172.21.15.136:6800/3380174081 conn(0x7fa0d806da80 0x7fa0d806ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.419+0000 7fa0e27fc700 1 --2- 172.21.15.136:0/3511325115 >> v2:172.21.15.164:6800/952209544 conn(0x7fa0d8077320 0x7fa0d80797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.419+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 --> v2:172.21.15.164:6800/952209544 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0d8079e50 con 0x7fa0d8077320 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.419+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0e808f5f0 con 0x7fa0ec100f80 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.419+0000 7fa0f0d4e700 1 --2- 172.21.15.136:0/3511325115 >> v2:172.21.15.136:6800/3380174081 conn(0x7fa0d806da80 0x7fa0d806ff40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0784b0 tx=0x7fa0dc00a380 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:28.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.419+0000 7fa0f1d50700 1 --2- 172.21.15.136:0/3511325115 >> v2:172.21.15.164:6800/952209544 conn(0x7fa0d8077320 0x7fa0d80797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:28.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.419+0000 7fa0f1d50700 1 --2- 172.21.15.136:0/3511325115 >> v2:172.21.15.164:6800/952209544 conn(0x7fa0d8077320 0x7fa0d80797c0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:28.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.420+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 <== osd.4 v2:172.21.15.164:6800/952209544 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa0d8079e50 con 0x7fa0d8077320 2024-03-26T05:22:28.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.436+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 --> v2:172.21.15.164:6800/952209544 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa0ec105fd0 con 0x7fa0d8077320 2024-03-26T05:22:28.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0e27fc700 1 -- 172.21.15.136:0/3511325115 <== osd.4 v2:172.21.15.164:6800/952209544 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa0ec105fd0 con 0x7fa0d8077320 2024-03-26T05:22:28.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 >> v2:172.21.15.164:6800/952209544 conn(0x7fa0d8077320 msgr2=0x7fa0d80797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:28.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 --2- 172.21.15.136:0/3511325115 >> v2:172.21.15.164:6800/952209544 conn(0x7fa0d8077320 0x7fa0d80797c0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:28.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 >> v2:172.21.15.136:6800/3380174081 conn(0x7fa0d806da80 msgr2=0x7fa0d806ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:28.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 --2- 172.21.15.136:0/3511325115 >> v2:172.21.15.136:6800/3380174081 conn(0x7fa0d806da80 0x7fa0d806ff40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0784b0 tx=0x7fa0dc00a380 comp rx=0 tx=0).stop 2024-03-26T05:22:28.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ec100f80 msgr2=0x7fa0ec102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:28.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 --2- 172.21.15.136:0/3511325115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ec100f80 0x7fa0ec102c30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80059d0 tx=0x7fa0e8004730 comp rx=0 tx=0).stop 2024-03-26T05:22:28.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f1d50700 1 -- 172.21.15.136:0/3511325115 reap_dead start 2024-03-26T05:22:28.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 shutdown_connections 2024-03-26T05:22:28.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 >> 172.21.15.136:0/3511325115 conn(0x7fa0ec077eb0 msgr2=0x7fa0ec069fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:28.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 shutdown_connections 2024-03-26T05:22:28.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:28.437+0000 7fa0f37b3700 1 -- 172.21.15.136:0/3511325115 wait complete. 2024-03-26T05:22:29.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:29 smithi136 ceph-mon[38106]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:29.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:29 smithi136 ceph-mon[29560]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:29.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:29 smithi164 ceph-mon[33554]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:30.715 INFO:teuthology.orchestra.run.smithi136.stdout:128849018911 2024-03-26T05:22:30.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T05:22:31.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 -- 172.21.15.136:0/17225166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 msgr2=0x7f942010d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 --2- 172.21.15.136:0/17225166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 0x7f942010d5c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f941800b0d0 tx=0x7f941800b3e0 comp rx=0 tx=0).stop 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 -- 172.21.15.136:0/17225166 shutdown_connections 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 --2- 172.21.15.136:0/17225166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94201067a0 0x7f9420106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 --2- 172.21.15.136:0/17225166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 0x7f942010d5c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 --2- 172.21.15.136:0/17225166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94201009c0 0x7f9420100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 -- 172.21.15.136:0/17225166 >> 172.21.15.136:0/17225166 conn(0x7f94200762e0 msgr2=0x7f94200766f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 -- 172.21.15.136:0/17225166 shutdown_connections 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 -- 172.21.15.136:0/17225166 wait complete. 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 Processor -- start 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.236+0000 7f9426deb700 1 -- start start 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9426deb700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94201009c0 0x7f9420196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:31.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9426deb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 0x7f9420196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9426deb700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94201067a0 0x7f9420197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9426deb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94201a61f0 con 0x7f9420101380 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9426deb700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94201a6360 con 0x7f94201067a0 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9426deb700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94201a6520 con 0x7f94201009c0 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9424b87700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94201009c0 0x7f9420196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 0x7f9420196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 0x7f9420196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38502/0 (socket says 172.21.15.136:38502) 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9424b87700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94201009c0 0x7f9420196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:50380/0 (socket says 172.21.15.136:50380) 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 -- 172.21.15.136:0/3375116299 learned_addr learned my addr 172.21.15.136:0/3375116299 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 -- 172.21.15.136:0/3375116299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94201009c0 msgr2=0x7f9420196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 --2- 172.21.15.136:0/3375116299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94201009c0 0x7f9420196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:31.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 -- 172.21.15.136:0/3375116299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94201067a0 msgr2=0x7f9420197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 --2- 172.21.15.136:0/3375116299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94201067a0 0x7f9420197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 -- 172.21.15.136:0/3375116299 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c009520 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9424b87700 1 --2- 172.21.15.136:0/3375116299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94201009c0 0x7f9420196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941ffff700 1 --2- 172.21.15.136:0/3375116299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 0x7f9420196ff0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f940c00eb30 tx=0x7f940c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c009880 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9418009d00 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f940c0044d0 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.237+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c003e20 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.238+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94201a6b70 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.238+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94201101d0 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.239+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f940c003f80 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.239+0000 7f941dffb700 1 --2- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6800/3380174081 conn(0x7f941006da80 0x7f941006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.239+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f940c014070 con 0x7f9420101380 2024-03-26T05:22:31.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.239+0000 7f9424b87700 1 --2- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6800/3380174081 conn(0x7f941006da80 0x7f941006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.239+0000 7f941dffb700 1 --2- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6805/1223127568 conn(0x7f9410077380 0x7f9410079820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.239+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 --> v2:172.21.15.136:6805/1223127568 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9410079eb0 con 0x7f9410077380 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.239+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f940c08fce0 con 0x7f9420101380 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.240+0000 7f9425388700 1 --2- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6805/1223127568 conn(0x7f9410077380 0x7f9410079820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.240+0000 7f9424b87700 1 --2- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6800/3380174081 conn(0x7f941006da80 0x7f941006ff40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9414009de0 tx=0x7f9414009190 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.240+0000 7f9425388700 1 --2- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6805/1223127568 conn(0x7f9410077380 0x7f9410079820 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.241+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 <== osd.1 v2:172.21.15.136:6805/1223127568 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9410079eb0 con 0x7f9410077380 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.257+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 --> v2:172.21.15.136:6805/1223127568 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9420104450 con 0x7f9410077380 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.257+0000 7f941dffb700 1 -- 172.21.15.136:0/3375116299 <== osd.1 v2:172.21.15.136:6805/1223127568 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9420104450 con 0x7f9410077380 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6805/1223127568 conn(0x7f9410077380 msgr2=0x7f9410079820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 --2- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6805/1223127568 conn(0x7f9410077380 0x7f9410079820 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6800/3380174081 conn(0x7f941006da80 msgr2=0x7f941006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 --2- 172.21.15.136:0/3375116299 >> v2:172.21.15.136:6800/3380174081 conn(0x7f941006da80 0x7f941006ff40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9414009de0 tx=0x7f9414009190 comp rx=0 tx=0).stop 2024-03-26T05:22:31.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 msgr2=0x7f9420196ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:31.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 --2- 172.21.15.136:0/3375116299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420101380 0x7f9420196ff0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f940c00eb30 tx=0x7f940c00ee40 comp rx=0 tx=0).stop 2024-03-26T05:22:31.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9425388700 1 -- 172.21.15.136:0/3375116299 reap_dead start 2024-03-26T05:22:31.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 shutdown_connections 2024-03-26T05:22:31.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 >> 172.21.15.136:0/3375116299 conn(0x7f94200762e0 msgr2=0x7f94200fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:31.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 shutdown_connections 2024-03-26T05:22:31.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:31.258+0000 7f9426deb700 1 -- 172.21.15.136:0/3375116299 wait complete. 2024-03-26T05:22:31.282 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:31 smithi136 ceph-mon[29560]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:31.282 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:31 smithi136 ceph-mon[38106]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:31.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:31 smithi164 ceph-mon[33554]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:32.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:22:31 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:22:32.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 -- 172.21.15.136:0/4264575240 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3988101bc0 msgr2=0x7f3988105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 --2- 172.21.15.136:0/4264575240 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3988101bc0 0x7f3988105b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f397c0098f0 tx=0x7f397c009c00 comp rx=0 tx=0).stop 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 -- 172.21.15.136:0/4264575240 shutdown_connections 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 --2- 172.21.15.136:0/4264575240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3988101210 0x7f39881015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 --2- 172.21.15.136:0/4264575240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3988106260 0x7f398810de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 --2- 172.21.15.136:0/4264575240 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3988101bc0 0x7f3988105b80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 -- 172.21.15.136:0/4264575240 >> 172.21.15.136:0/4264575240 conn(0x7f3988076fd0 msgr2=0x7f39880773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 -- 172.21.15.136:0/4264575240 shutdown_connections 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.365+0000 7f398e02b700 1 -- 172.21.15.136:0/4264575240 wait complete. 2024-03-26T05:22:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f398e02b700 1 Processor -- start 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f398e02b700 1 -- start start 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f398e02b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3988101210 0x7f398819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f398e02b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3988101bc0 0x7f398819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f398e02b700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3988106260 0x7f3988195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f398e02b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3988196080 con 0x7f3988101bc0 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f398e02b700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39881961c0 con 0x7f3988101210 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f398e02b700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3988196380 con 0x7f3988106260 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f3986ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3988101bc0 0x7f398819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f3987fff700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3988106260 0x7f3988195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f3986ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3988101bc0 0x7f398819bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38530/0 (socket says 172.21.15.136:38530) 2024-03-26T05:22:32.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f3987fff700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3988106260 0x7f3988195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:50400/0 (socket says 172.21.15.136:50400) 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f3986ffd700 1 -- 172.21.15.136:0/2919783136 learned_addr learned my addr 172.21.15.136:0/2919783136 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.366+0000 7f39877fe700 1 --2- 172.21.15.136:0/2919783136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3988101210 0x7f398819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3986ffd700 1 -- 172.21.15.136:0/2919783136 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3988106260 msgr2=0x7f3988195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3986ffd700 1 --2- 172.21.15.136:0/2919783136 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3988106260 0x7f3988195a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3986ffd700 1 -- 172.21.15.136:0/2919783136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3988101210 msgr2=0x7f398819b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3986ffd700 1 --2- 172.21.15.136:0/2919783136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3988101210 0x7f398819b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3986ffd700 1 -- 172.21.15.136:0/2919783136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3980009d00 con 0x7f3988101bc0 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3986ffd700 1 --2- 172.21.15.136:0/2919783136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3988101bc0 0x7f398819bd50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f397c009230 tx=0x7f397c00f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c024070 con 0x7f3988101bc0 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f397c00ba90 con 0x7f3988101bc0 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c009520 con 0x7f3988101bc0 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c0038e0 con 0x7f3988101bc0 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.367+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3988196880 con 0x7f3988101bc0 2024-03-26T05:22:32.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.368+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f398810f8e0 con 0x7f3988101bc0 2024-03-26T05:22:32.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.369+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f397c028030 con 0x7f3988101bc0 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.369+0000 7f3984ff9700 1 --2- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6800/3380174081 conn(0x7f397006da70 0x7f397006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.369+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f397c0970e0 con 0x7f3988101bc0 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.369+0000 7f39877fe700 1 --2- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6800/3380174081 conn(0x7f397006da70 0x7f397006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.369+0000 7f3984ff9700 1 --2- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6813/802386900 conn(0x7f3970077370 0x7f3970079810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.370+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 --> v2:172.21.15.136:6813/802386900 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3970079ea0 con 0x7f3970077370 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.370+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f397c097530 con 0x7f3988101bc0 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.370+0000 7f3987fff700 1 --2- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6813/802386900 conn(0x7f3970077370 0x7f3970079810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.370+0000 7f3987fff700 1 --2- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6813/802386900 conn(0x7f3970077370 0x7f3970079810 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.370+0000 7f39877fe700 1 --2- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6800/3380174081 conn(0x7f397006da70 0x7f397006ff30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3988105d10 tx=0x7f3978009190 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.371+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 <== osd.3 v2:172.21.15.136:6813/802386900 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3970079ea0 con 0x7f3970077370 2024-03-26T05:22:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.393+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 --> v2:172.21.15.136:6813/802386900 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39880ff780 con 0x7f3970077370 2024-03-26T05:22:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.393+0000 7f3984ff9700 1 -- 172.21.15.136:0/2919783136 <== osd.3 v2:172.21.15.136:6813/802386900 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f39880ff780 con 0x7f3970077370 2024-03-26T05:22:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.393+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6813/802386900 conn(0x7f3970077370 msgr2=0x7f3970079810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.393+0000 7f398e02b700 1 --2- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6813/802386900 conn(0x7f3970077370 0x7f3970079810 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6800/3380174081 conn(0x7f397006da70 msgr2=0x7f397006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f398e02b700 1 --2- 172.21.15.136:0/2919783136 >> v2:172.21.15.136:6800/3380174081 conn(0x7f397006da70 0x7f397006ff30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3988105d10 tx=0x7f3978009190 comp rx=0 tx=0).stop 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3988101bc0 msgr2=0x7f398819bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f398e02b700 1 --2- 172.21.15.136:0/2919783136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3988101bc0 0x7f398819bd50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f397c009230 tx=0x7f397c00f9e0 comp rx=0 tx=0).stop 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f3987fff700 1 -- 172.21.15.136:0/2919783136 reap_dead start 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 shutdown_connections 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 >> 172.21.15.136:0/2919783136 conn(0x7f3988076fd0 msgr2=0x7f39880fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 shutdown_connections 2024-03-26T05:22:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:32.394+0000 7f398e02b700 1 -- 172.21.15.136:0/2919783136 wait complete. 2024-03-26T05:22:33.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:33 smithi136 ceph-mon[38106]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:33.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:33.488 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:33 smithi136 ceph-mon[29560]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:33.488 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:33.517 INFO:teuthology.orchestra.run.smithi136.stdout:163208757274 2024-03-26T05:22:33.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T05:22:33.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:33 smithi164 ceph-mon[33554]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:33.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:34.363 INFO:teuthology.orchestra.run.smithi136.stdout:55834574895 2024-03-26T05:22:34.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T05:22:34.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 -- 172.21.15.136:0/32083650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 msgr2=0x7f30781073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:34.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 --2- 172.21.15.136:0/32083650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 0x7f30781073a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f306c009990 tx=0x7f306c009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 -- 172.21.15.136:0/32083650 shutdown_connections 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 --2- 172.21.15.136:0/32083650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3078101920 0x7f307810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 --2- 172.21.15.136:0/32083650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 0x7f30781073a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 --2- 172.21.15.136:0/32083650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3078100f80 0x7f30781013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 -- 172.21.15.136:0/32083650 >> 172.21.15.136:0/32083650 conn(0x7f3078077eb0 msgr2=0x7f30780782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 -- 172.21.15.136:0/32083650 shutdown_connections 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.861+0000 7f307d457700 1 -- 172.21.15.136:0/32083650 wait complete. 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f307d457700 1 Processor -- start 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f307d457700 1 -- start start 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f307d457700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3078100f80 0x7f3078197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f307d457700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3078101920 0x7f3078198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f307d457700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 0x7f307819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f307d457700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307819cc30 con 0x7f3078101920 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f307d457700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307819cda0 con 0x7f3078100f80 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f307d457700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307819cf60 con 0x7f3078106fc0 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30767fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3078101920 0x7f3078198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30777fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 0x7f307819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30777fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 0x7f307819c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:50412/0 (socket says 172.21.15.136:50412) 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30777fe700 1 -- 172.21.15.136:0/251187228 learned_addr learned my addr 172.21.15.136:0/251187228 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30777fe700 1 -- 172.21.15.136:0/251187228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3078100f80 msgr2=0x7f3078197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30777fe700 1 --2- 172.21.15.136:0/251187228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3078100f80 0x7f3078197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30777fe700 1 -- 172.21.15.136:0/251187228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3078101920 msgr2=0x7f3078198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:34.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30777fe700 1 --2- 172.21.15.136:0/251187228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3078101920 0x7f3078198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:34.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30777fe700 1 -- 172.21.15.136:0/251187228 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068009520 con 0x7f3078106fc0 2024-03-26T05:22:34.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.862+0000 7f30767fc700 1 --2- 172.21.15.136:0/251187228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3078101920 0x7f3078198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:34.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.863+0000 7f30777fe700 1 --2- 172.21.15.136:0/251187228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 0x7f307819c620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f307000b970 tx=0x7f307000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:34.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.863+0000 7f305ffff700 1 -- 172.21.15.136:0/251187228 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307000c930 con 0x7f3078106fc0 2024-03-26T05:22:34.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.863+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306c0095f0 con 0x7f3078106fc0 2024-03-26T05:22:34.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.863+0000 7f305ffff700 1 -- 172.21.15.136:0/251187228 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3070014440 con 0x7f3078106fc0 2024-03-26T05:22:34.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.863+0000 7f305ffff700 1 -- 172.21.15.136:0/251187228 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30700075d0 con 0x7f3078106fc0 2024-03-26T05:22:34.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.863+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30781a48d0 con 0x7f3078106fc0 2024-03-26T05:22:34.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.864+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30781094e0 con 0x7f3078106fc0 2024-03-26T05:22:34.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.864+0000 7f305ffff700 1 -- 172.21.15.136:0/251187228 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f307000ca90 con 0x7f3078106fc0 2024-03-26T05:22:34.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.865+0000 7f305ffff700 1 --2- 172.21.15.136:0/251187228 >> v2:172.21.15.136:6800/3380174081 conn(0x7f306006da80 0x7f306006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:34.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.865+0000 7f305ffff700 1 -- 172.21.15.136:0/251187228 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f3070006e40 con 0x7f3078106fc0 2024-03-26T05:22:34.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.865+0000 7f3076ffd700 1 --2- 172.21.15.136:0/251187228 >> v2:172.21.15.136:6800/3380174081 conn(0x7f306006da80 0x7f306006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:34.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.865+0000 7f3076ffd700 1 --2- 172.21.15.136:0/251187228 >> v2:172.21.15.136:6800/3380174081 conn(0x7f306006da80 0x7f306006ff40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f306c00b7d0 tx=0x7f306c005fb0 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:34.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:34.870+0000 7f305ffff700 1 -- 172.21.15.136:0/251187228 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3070059fe0 con 0x7f3078106fc0 2024-03-26T05:22:35.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.068+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f30781990b0 con 0x7f3078106fc0 2024-03-26T05:22:35.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.068+0000 7f305ffff700 1 -- 172.21.15.136:0/251187228 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3070059b70 con 0x7f3078106fc0 2024-03-26T05:22:35.059 INFO:teuthology.orchestra.run.smithi136.stdout:197568495637 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 >> v2:172.21.15.136:6800/3380174081 conn(0x7f306006da80 msgr2=0x7f306006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 --2- 172.21.15.136:0/251187228 >> v2:172.21.15.136:6800/3380174081 conn(0x7f306006da80 0x7f306006ff40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f306c00b7d0 tx=0x7f306c005fb0 comp rx=0 tx=0).stop 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 msgr2=0x7f307819c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 --2- 172.21.15.136:0/251187228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 0x7f307819c620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f307000b970 tx=0x7f307000bc80 comp rx=0 tx=0).stop 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 shutdown_connections 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 --2- 172.21.15.136:0/251187228 >> v2:172.21.15.136:6800/3380174081 conn(0x7f306006da80 0x7f306006ff40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 --2- 172.21.15.136:0/251187228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3078100f80 0x7f3078197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 --2- 172.21.15.136:0/251187228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3078106fc0 0x7f307819c620 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 --2- 172.21.15.136:0/251187228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3078101920 0x7f3078198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:35.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.070+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 >> 172.21.15.136:0/251187228 conn(0x7f3078077eb0 msgr2=0x7f30780ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:35.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.071+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 shutdown_connections 2024-03-26T05:22:35.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:35.071+0000 7f307d457700 1 -- 172.21.15.136:0/251187228 wait complete. 2024-03-26T05:22:35.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:35 smithi136 ceph-mon[38106]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:35.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/251187228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:22:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:35 smithi136 ceph-mon[29560]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:35.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/251187228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:22:35.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:35 smithi164 ceph-mon[33554]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:35.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/251187228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T05:22:35.939 INFO:teuthology.orchestra.run.smithi136.stdout:107374182437 2024-03-26T05:22:35.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T05:22:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:36.079 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:36.694 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495637 for osd.6 2024-03-26T05:22:36.695 DEBUG:teuthology.parallel:result is None 2024-03-26T05:22:37.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 -- 172.21.15.136:0/3536038868 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae4107440 msgr2=0x7ffae410efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:37.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/3536038868 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae4107440 0x7ffae410efc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffad4009990 tx=0x7ffad4009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 -- 172.21.15.136:0/3536038868 shutdown_connections 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/3536038868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffae4100c20 0x7ffae4101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/3536038868 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae4107440 0x7ffae410efc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/3536038868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffae41015d0 0x7ffae4106d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 -- 172.21.15.136:0/3536038868 >> 172.21.15.136:0/3536038868 conn(0x7ffae40762e0 msgr2=0x7ffae40766f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 -- 172.21.15.136:0/3536038868 shutdown_connections 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.269+0000 7ffaeaca4700 1 -- 172.21.15.136:0/3536038868 wait complete. 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffaeaca4700 1 Processor -- start 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffaeaca4700 1 -- start start 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffaeaca4700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffae4100c20 0x7ffae419da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffaeaca4700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae41015d0 0x7ffae419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffaeaca4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffae4107440 0x7ffae4197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffaeaca4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae41982c0 con 0x7ffae4107440 2024-03-26T05:22:37.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffaeaca4700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae4198430 con 0x7ffae4100c20 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffaeaca4700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae41985f0 con 0x7ffae41015d0 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffae9241700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffae4107440 0x7ffae4197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffae9241700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffae4107440 0x7ffae4197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38570/0 (socket says 172.21.15.136:38570) 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffae9241700 1 -- 172.21.15.136:0/4042604905 learned_addr learned my addr 172.21.15.136:0/4042604905 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffae3fff700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae41015d0 0x7ffae419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.270+0000 7ffae8a40700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffae4100c20 0x7ffae419da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae9241700 1 -- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae41015d0 msgr2=0x7ffae419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae9241700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae41015d0 0x7ffae419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae9241700 1 -- 172.21.15.136:0/4042604905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffae4100c20 msgr2=0x7ffae419da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae9241700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffae4100c20 0x7ffae419da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae9241700 1 -- 172.21.15.136:0/4042604905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad8009520 con 0x7ffae4107440 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae8a40700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffae4100c20 0x7ffae419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae3fff700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae41015d0 0x7ffae419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:37.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae9241700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffae4107440 0x7ffae4197b10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffad4005f50 tx=0x7ffad4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:37.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae1ffb700 1 -- 172.21.15.136:0/4042604905 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad401d070 con 0x7ffae4107440 2024-03-26T05:22:37.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad40095f0 con 0x7ffae4107440 2024-03-26T05:22:37.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae1ffb700 1 -- 172.21.15.136:0/4042604905 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ffad400b610 con 0x7ffae4107440 2024-03-26T05:22:37.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffae1ffb700 1 -- 172.21.15.136:0/4042604905 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad40173f0 con 0x7ffae4107440 2024-03-26T05:22:37.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.271+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae41a47f0 con 0x7ffae4107440 2024-03-26T05:22:37.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.272+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae406a2f0 con 0x7ffae4107440 2024-03-26T05:22:37.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.273+0000 7ffae1ffb700 1 -- 172.21.15.136:0/4042604905 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7ffad400fb10 con 0x7ffae4107440 2024-03-26T05:22:37.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.273+0000 7ffae1ffb700 1 --2- 172.21.15.136:0/4042604905 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffacc06da80 0x7ffacc06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:37.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.273+0000 7ffae8a40700 1 --2- 172.21.15.136:0/4042604905 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffacc06da80 0x7ffacc06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:37.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.273+0000 7ffae1ffb700 1 -- 172.21.15.136:0/4042604905 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7ffad408e910 con 0x7ffae4107440 2024-03-26T05:22:37.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.273+0000 7ffae8a40700 1 --2- 172.21.15.136:0/4042604905 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffacc06da80 0x7ffacc06ff40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009fa0 tx=0x7ffadc009570 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:37.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.278+0000 7ffae1ffb700 1 -- 172.21.15.136:0/4042604905 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffad405bed0 con 0x7ffae4107440 2024-03-26T05:22:37.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.485+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ffae4198fd0 con 0x7ffae4107440 2024-03-26T05:22:37.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.485+0000 7ffae1ffb700 1 -- 172.21.15.136:0/4042604905 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffad4027020 con 0x7ffae4107440 2024-03-26T05:22:37.476 INFO:teuthology.orchestra.run.smithi136.stdout:77309411371 2024-03-26T05:22:37.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffacc06da80 msgr2=0x7ffacc06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/4042604905 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffacc06da80 0x7ffacc06ff40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009fa0 tx=0x7ffadc009570 comp rx=0 tx=0).stop 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffae4107440 msgr2=0x7ffae4197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffae4107440 0x7ffae4197b10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffad4005f50 tx=0x7ffad4005740 comp rx=0 tx=0).stop 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 shutdown_connections 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/4042604905 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffacc06da80 0x7ffacc06ff40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffae4100c20 0x7ffae419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffae4107440 0x7ffae4197b10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 --2- 172.21.15.136:0/4042604905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffae41015d0 0x7ffae419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 >> 172.21.15.136:0/4042604905 conn(0x7ffae40762e0 msgr2=0x7ffae40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 shutdown_connections 2024-03-26T05:22:37.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:37.487+0000 7ffaeaca4700 1 -- 172.21.15.136:0/4042604905 wait complete. 2024-03-26T05:22:37.482 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:37 smithi136 ceph-mon[38106]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:37.482 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:37 smithi136 ceph-mon[29560]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:37.482 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:22:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:37.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:37 smithi164 ceph-mon[33554]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:37.916 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:37.916 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:37.916 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:38.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4042604905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:22:38.514 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4042604905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:22:38.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4042604905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T05:22:39.471 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:39 smithi136 ceph-mon[38106]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:39.472 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:39 smithi136 ceph-mon[29560]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:39.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:39 smithi164 ceph-mon[33554]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:40.530 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:40 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:40.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:40 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:40.557 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411371 for osd.2 2024-03-26T05:22:40.557 DEBUG:teuthology.parallel:result is None 2024-03-26T05:22:40.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:40 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:41 smithi136 ceph-mon[38106]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:41.396 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:41 smithi136 ceph-mon[29560]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:41.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:41 smithi164 ceph-mon[33554]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:42.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:22:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:22:43.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:43 smithi136 ceph-mon[38106]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:43.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:43.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:43 smithi136 ceph-mon[29560]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:43.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:43.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:43 smithi164 ceph-mon[33554]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:43.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:44.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 -- 172.21.15.136:0/3348491930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 msgr2=0x7f33e810de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:44.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 --2- 172.21.15.136:0/3348491930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 0x7f33e810de50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f33e000b0a0 tx=0x7f33e000b3b0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 -- 172.21.15.136:0/3348491930 shutdown_connections 2024-03-26T05:22:44.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 --2- 172.21.15.136:0/3348491930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e8107170 0x7f33e8107550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 --2- 172.21.15.136:0/3348491930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 0x7f33e810de50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 --2- 172.21.15.136:0/3348491930 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33e8101260 0x7f33e81016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 -- 172.21.15.136:0/3348491930 >> 172.21.15.136:0/3348491930 conn(0x7f33e8077de0 msgr2=0x7f33e80781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:44.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 -- 172.21.15.136:0/3348491930 shutdown_connections 2024-03-26T05:22:44.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.496+0000 7f33ed961700 1 -- 172.21.15.136:0/3348491930 wait complete. 2024-03-26T05:22:44.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33ed961700 1 Processor -- start 2024-03-26T05:22:44.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33ed961700 1 -- start start 2024-03-26T05:22:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33ed961700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33e8101260 0x7f33e8197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33ed961700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 0x7f33e81982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33ed961700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e8107170 0x7f33e819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33ed961700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e819cc90 con 0x7f33e8101c20 2024-03-26T05:22:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33ed961700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e819ce00 con 0x7f33e8107170 2024-03-26T05:22:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33ed961700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e819cfc0 con 0x7f33e8101260 2024-03-26T05:22:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33e6ffd700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33e8101260 0x7f33e8197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.497+0000 7f33e67fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 0x7f33e81982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e67fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 0x7f33e81982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56688/0 (socket says 172.21.15.136:56688) 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e6ffd700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33e8101260 0x7f33e8197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:59100/0 (socket says 172.21.15.136:59100) 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e67fc700 1 -- 172.21.15.136:0/1958448759 learned_addr learned my addr 172.21.15.136:0/1958448759 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e77fe700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e8107170 0x7f33e819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e67fc700 1 -- 172.21.15.136:0/1958448759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33e8101260 msgr2=0x7f33e8197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e67fc700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33e8101260 0x7f33e8197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e67fc700 1 -- 172.21.15.136:0/1958448759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e8107170 msgr2=0x7f33e819c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e67fc700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e8107170 0x7f33e819c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e67fc700 1 -- 172.21.15.136:0/1958448759 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc009520 con 0x7f33e8101c20 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e6ffd700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33e8101260 0x7f33e8197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33e67fc700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 0x7f33e81982f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00eb30 tx=0x7f33dc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33ec95f700 1 -- 172.21.15.136:0/1958448759 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc009880 con 0x7f33e8101c20 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33ec95f700 1 -- 172.21.15.136:0/1958448759 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f33dc0044d0 con 0x7f33e8101c20 2024-03-26T05:22:44.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e0009d00 con 0x7f33e8101c20 2024-03-26T05:22:44.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33ec95f700 1 -- 172.21.15.136:0/1958448759 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc003d10 con 0x7f33e8101c20 2024-03-26T05:22:44.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.498+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e81a4930 con 0x7f33e8101c20 2024-03-26T05:22:44.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.499+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e8102700 con 0x7f33e8101c20 2024-03-26T05:22:44.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.500+0000 7f33ec95f700 1 -- 172.21.15.136:0/1958448759 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f33dc003e90 con 0x7f33e8101c20 2024-03-26T05:22:44.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.500+0000 7f33ec95f700 1 --2- 172.21.15.136:0/1958448759 >> v2:172.21.15.136:6800/3380174081 conn(0x7f33d006da80 0x7f33d006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:44.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.500+0000 7f33ec95f700 1 -- 172.21.15.136:0/1958448759 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f33dc014070 con 0x7f33e8101c20 2024-03-26T05:22:44.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.500+0000 7f33e6ffd700 1 --2- 172.21.15.136:0/1958448759 >> v2:172.21.15.136:6800/3380174081 conn(0x7f33d006da80 0x7f33d006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:44.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.501+0000 7f33e6ffd700 1 --2- 172.21.15.136:0/1958448759 >> v2:172.21.15.136:6800/3380174081 conn(0x7f33d006da80 0x7f33d006ff40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33d8005950 tx=0x7f33d800a300 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:44.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.505+0000 7f33ec95f700 1 -- 172.21.15.136:0/1958448759 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f33dc05abf0 con 0x7f33e8101c20 2024-03-26T05:22:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.702+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f33e8199110 con 0x7f33e8101c20 2024-03-26T05:22:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.702+0000 7f33ec95f700 1 -- 172.21.15.136:0/1958448759 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f33dc05a780 con 0x7f33e8101c20 2024-03-26T05:22:44.693 INFO:teuthology.orchestra.run.smithi136.stdout:231928234003 2024-03-26T05:22:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 >> v2:172.21.15.136:6800/3380174081 conn(0x7f33d006da80 msgr2=0x7f33d006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 --2- 172.21.15.136:0/1958448759 >> v2:172.21.15.136:6800/3380174081 conn(0x7f33d006da80 0x7f33d006ff40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33d8005950 tx=0x7f33d800a300 comp rx=0 tx=0).stop 2024-03-26T05:22:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 msgr2=0x7f33e81982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 0x7f33e81982f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00eb30 tx=0x7f33dc00ee40 comp rx=0 tx=0).stop 2024-03-26T05:22:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 shutdown_connections 2024-03-26T05:22:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 --2- 172.21.15.136:0/1958448759 >> v2:172.21.15.136:6800/3380174081 conn(0x7f33d006da80 0x7f33d006ff40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e8107170 0x7f33e819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33e8101260 0x7f33e8197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 --2- 172.21.15.136:0/1958448759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33e8101c20 0x7f33e81982f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:44.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 >> 172.21.15.136:0/1958448759 conn(0x7f33e8077de0 msgr2=0x7f33e80ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:44.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 shutdown_connections 2024-03-26T05:22:44.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:44.705+0000 7f33ed961700 1 -- 172.21.15.136:0/1958448759 wait complete. 2024-03-26T05:22:45.410 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:45 smithi136 ceph-mon[38106]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:45.410 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1958448759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:22:45.410 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:45 smithi136 ceph-mon[29560]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:45.410 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1958448759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:22:45.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:45 smithi164 ceph-mon[33554]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:45.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1958448759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T05:22:46.324 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 podman[71855]: 2024-03-26 05:22:46.209183861 +0000 UTC m=+0.876180203 container create 8c131f9b360d36befa0994e7d5ee9cded3f0ab53e7536c6ef1817d2353f5f529 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:22:46.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 -- 172.21.15.136:0/2643752263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154101240 msgr2=0x7ff154101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:46.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 --2- 172.21.15.136:0/2643752263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154101240 0x7ff154101620 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff144009990 tx=0x7ff144009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:46.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 -- 172.21.15.136:0/2643752263 shutdown_connections 2024-03-26T05:22:46.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 --2- 172.21.15.136:0/2643752263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff154106290 0x7ff15410de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:46.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 --2- 172.21.15.136:0/2643752263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154101240 0x7ff154101620 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:46.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 --2- 172.21.15.136:0/2643752263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff154101bf0 0x7ff154105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:46.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 -- 172.21.15.136:0/2643752263 >> 172.21.15.136:0/2643752263 conn(0x7ff154076fd0 msgr2=0x7ff1540773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 -- 172.21.15.136:0/2643752263 shutdown_connections 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.967+0000 7ff159d9c700 1 -- 172.21.15.136:0/2643752263 wait complete. 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.968+0000 7ff159d9c700 1 Processor -- start 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.968+0000 7ff159d9c700 1 -- start start 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.968+0000 7ff159d9c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff154101240 0x7ff15419da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.968+0000 7ff159d9c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff154101bf0 0x7ff15419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.968+0000 7ff159d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154106290 0x7ff154197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.968+0000 7ff159d9c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff154198310 con 0x7ff154106290 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.968+0000 7ff159d9c700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff154198480 con 0x7ff154101bf0 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.968+0000 7ff159d9c700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff154198640 con 0x7ff154101240 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154106290 0x7ff154197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff1537fe700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff154101240 0x7ff15419da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154106290 0x7ff154197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56694/0 (socket says 172.21.15.136:56694) 2024-03-26T05:22:46.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 -- 172.21.15.136:0/1785224596 learned_addr learned my addr 172.21.15.136:0/1785224596 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff152ffd700 1 --2- 172.21.15.136:0/1785224596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff154101bf0 0x7ff15419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 -- 172.21.15.136:0/1785224596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff154101240 msgr2=0x7ff15419da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 --2- 172.21.15.136:0/1785224596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff154101240 0x7ff15419da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 -- 172.21.15.136:0/1785224596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff154101bf0 msgr2=0x7ff15419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 --2- 172.21.15.136:0/1785224596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff154101bf0 0x7ff15419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 -- 172.21.15.136:0/1785224596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff148009520 con 0x7ff154106290 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff153fff700 1 --2- 172.21.15.136:0/1785224596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154106290 0x7ff154197b60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00b970 tx=0x7ff14c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff150ff9700 1 -- 172.21.15.136:0/1785224596 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c00d860 con 0x7ff154106290 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1440095f0 con 0x7ff154106290 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff150ff9700 1 -- 172.21.15.136:0/1785224596 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ff14c00de70 con 0x7ff154106290 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff150ff9700 1 -- 172.21.15.136:0/1785224596 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c017400 con 0x7ff154106290 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.969+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1541a49c0 con 0x7ff154106290 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.970+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154069f90 con 0x7ff154106290 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.971+0000 7ff150ff9700 1 -- 172.21.15.136:0/1785224596 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7ff14c00d9c0 con 0x7ff154106290 2024-03-26T05:22:46.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.971+0000 7ff150ff9700 1 --2- 172.21.15.136:0/1785224596 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff13c06da80 0x7ff13c06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:46.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.971+0000 7ff1537fe700 1 --2- 172.21.15.136:0/1785224596 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff13c06da80 0x7ff13c06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:46.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.971+0000 7ff150ff9700 1 -- 172.21.15.136:0/1785224596 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7ff14c006e40 con 0x7ff154106290 2024-03-26T05:22:46.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.972+0000 7ff1537fe700 1 --2- 172.21.15.136:0/1785224596 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff13c06da80 0x7ff13c06ff40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1440098e0 tx=0x7ff144005960 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:46.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:46.976+0000 7ff150ff9700 1 -- 172.21.15.136:0/1785224596 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff14c05a480 con 0x7ff154106290 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 podman[71855]: 2024-03-26 05:22:46.764629143 +0000 UTC m=+1.431625444 container init 8c131f9b360d36befa0994e7d5ee9cded3f0ab53e7536c6ef1817d2353f5f529 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 podman[71855]: 2024-03-26 05:22:46.767305352 +0000 UTC m=+1.434301653 container start 8c131f9b360d36befa0994e7d5ee9cded3f0ab53e7536c6ef1817d2353f5f529 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:46.779Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:46.779Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:46.780Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.136 port=9094 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:46.781Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:46.803Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:46.804Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:46.806Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T05:22:47.065 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:46 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:46.806Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T05:22:47.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 -- 172.21.15.136:0/616374561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c101920 msgr2=0x7f446c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:47.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 --2- 172.21.15.136:0/616374561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c101920 0x7f446c10de00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f446400b0a0 tx=0x7f446400b3b0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 -- 172.21.15.136:0/616374561 shutdown_connections 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 --2- 172.21.15.136:0/616374561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f446c106fc0 0x7f446c1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 --2- 172.21.15.136:0/616374561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c101920 0x7f446c10de00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 --2- 172.21.15.136:0/616374561 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f446c100f80 0x7f446c1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 -- 172.21.15.136:0/616374561 >> 172.21.15.136:0/616374561 conn(0x7f446c077eb0 msgr2=0x7f446c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 -- 172.21.15.136:0/616374561 shutdown_connections 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.176+0000 7f4472ef2700 1 -- 172.21.15.136:0/616374561 wait complete. 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.177+0000 7f4472ef2700 1 Processor -- start 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.177+0000 7f4472ef2700 1 -- start start 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.177+0000 7f4472ef2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c100f80 0x7f446c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.177+0000 7f4472ef2700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f446c101920 0x7f446c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.177+0000 7f4472ef2700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f446c106fc0 0x7f446c196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.177+0000 7f4472ef2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c197130 con 0x7f446c100f80 2024-03-26T05:22:47.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.177+0000 7f4472ef2700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c197270 con 0x7f446c101920 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.177+0000 7f4472ef2700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446c197430 con 0x7f446c106fc0 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f4470c8e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c100f80 0x7f446c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f4470c8e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c100f80 0x7f446c19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56718/0 (socket says 172.21.15.136:56718) 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f4470c8e700 1 -- 172.21.15.136:0/4154002122 learned_addr learned my addr 172.21.15.136:0/4154002122 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f447148f700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f446c106fc0 0x7f446c196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f446bfff700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f446c101920 0x7f446c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f447148f700 1 -- 172.21.15.136:0/4154002122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f446c101920 msgr2=0x7f446c19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f447148f700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f446c101920 0x7f446c19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f447148f700 1 -- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c100f80 msgr2=0x7f446c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f447148f700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c100f80 0x7f446c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f447148f700 1 -- 172.21.15.136:0/4154002122 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458009520 con 0x7f446c106fc0 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f4470c8e700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c100f80 0x7f446c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f447148f700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f446c106fc0 0x7f446c196ad0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f446400b0a0 tx=0x7f446400be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:47.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f4469ffb700 1 -- 172.21.15.136:0/4154002122 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446400e040 con 0x7f446c106fc0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f4469ffb700 1 -- 172.21.15.136:0/4154002122 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f446401d470 con 0x7f446c106fc0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f4469ffb700 1 -- 172.21.15.136:0/4154002122 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44640052a0 con 0x7f446c106fc0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.178+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4464009d00 con 0x7f446c106fc0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.179+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c069d30 con 0x7f446c106fc0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.180+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c102670 con 0x7f446c106fc0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.180+0000 7f4469ffb700 1 -- 172.21.15.136:0/4154002122 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f44640074e0 con 0x7f446c106fc0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.180+0000 7f4469ffb700 1 --2- 172.21.15.136:0/4154002122 >> v2:172.21.15.136:6800/3380174081 conn(0x7f445c06da80 0x7f445c06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.180+0000 7f4469ffb700 1 -- 172.21.15.136:0/4154002122 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f446408e730 con 0x7f446c106fc0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.181+0000 7f4470c8e700 1 --2- 172.21.15.136:0/4154002122 >> v2:172.21.15.136:6800/3380174081 conn(0x7f445c06da80 0x7f445c06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:47.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.181+0000 7f4470c8e700 1 --2- 172.21.15.136:0/4154002122 >> v2:172.21.15.136:6800/3380174081 conn(0x7f445c06da80 0x7f445c06ff40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4460005950 tx=0x7f446000e040 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:47.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.184+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff154198fd0 con 0x7ff154106290 2024-03-26T05:22:47.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.184+0000 7ff150ff9700 1 -- 172.21.15.136:0/1785224596 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff14c05a010 con 0x7ff154106290 2024-03-26T05:22:47.176 INFO:teuthology.orchestra.run.smithi136.stdout:128849018915 2024-03-26T05:22:47.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.185+0000 7f4469ffb700 1 -- 172.21.15.136:0/4154002122 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f446405bbd0 con 0x7f446c106fc0 2024-03-26T05:22:47.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff13c06da80 msgr2=0x7ff13c06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 --2- 172.21.15.136:0/1785224596 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff13c06da80 0x7ff13c06ff40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1440098e0 tx=0x7ff144005960 comp rx=0 tx=0).stop 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154106290 msgr2=0x7ff154197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 --2- 172.21.15.136:0/1785224596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154106290 0x7ff154197b60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00b970 tx=0x7ff14c00bc80 comp rx=0 tx=0).stop 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 shutdown_connections 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 --2- 172.21.15.136:0/1785224596 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff13c06da80 0x7ff13c06ff40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 --2- 172.21.15.136:0/1785224596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff154106290 0x7ff154197b60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 --2- 172.21.15.136:0/1785224596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff154101240 0x7ff15419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 --2- 172.21.15.136:0/1785224596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff154101bf0 0x7ff15419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 >> 172.21.15.136:0/1785224596 conn(0x7ff154076fd0 msgr2=0x7ff154104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 shutdown_connections 2024-03-26T05:22:47.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.187+0000 7ff159d9c700 1 -- 172.21.15.136:0/1785224596 wait complete. 2024-03-26T05:22:47.322 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-mon[38106]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:47.322 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1785224596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:22:47.322 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:22:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:47.323 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:47 smithi136 bash[71855]: 8c131f9b360d36befa0994e7d5ee9cded3f0ab53e7536c6ef1817d2353f5f529 2024-03-26T05:22:47.323 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:47 smithi136 systemd[1]: Started Ceph alertmanager.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:22:47.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-mon[29560]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:47.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1785224596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:22:47.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.400+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f446c197fb0 con 0x7f446c106fc0 2024-03-26T05:22:47.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.400+0000 7f4469ffb700 1 -- 172.21.15.136:0/4154002122 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f446405b760 con 0x7f446c106fc0 2024-03-26T05:22:47.390 INFO:teuthology.orchestra.run.smithi136.stdout:163208757278 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 >> v2:172.21.15.136:6800/3380174081 conn(0x7f445c06da80 msgr2=0x7f445c06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 --2- 172.21.15.136:0/4154002122 >> v2:172.21.15.136:6800/3380174081 conn(0x7f445c06da80 0x7f445c06ff40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4460005950 tx=0x7f446000e040 comp rx=0 tx=0).stop 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f446c106fc0 msgr2=0x7f446c196ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f446c106fc0 0x7f446c196ad0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f446400b0a0 tx=0x7f446400be80 comp rx=0 tx=0).stop 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 shutdown_connections 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 --2- 172.21.15.136:0/4154002122 >> v2:172.21.15.136:6800/3380174081 conn(0x7f445c06da80 0x7f445c06ff40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f446c106fc0 0x7f446c196ad0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f446c100f80 0x7f446c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 --2- 172.21.15.136:0/4154002122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f446c101920 0x7f446c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:47.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.402+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 >> 172.21.15.136:0/4154002122 conn(0x7f446c077eb0 msgr2=0x7f446c0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:47.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.403+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 shutdown_connections 2024-03-26T05:22:47.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:47.403+0000 7f4472ef2700 1 -- 172.21.15.136:0/4154002122 wait complete. 2024-03-26T05:22:47.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:47] ENGINE Bus STOPPING 2024-03-26T05:22:47.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:22:47.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:47] ENGINE Bus STOPPED 2024-03-26T05:22:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:47] ENGINE Bus STARTING 2024-03-26T05:22:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:47] ENGINE Serving on http://:::9283 2024-03-26T05:22:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:22:47] ENGINE Bus STARTED 2024-03-26T05:22:47.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:47 smithi164 ceph-mon[33554]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:47.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1785224596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T05:22:48.214 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018911 got 128849018915 for osd.4 2024-03-26T05:22:48.214 DEBUG:teuthology.parallel:result is None 2024-03-26T05:22:48.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4154002122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[29560]: Deploying daemon grafana.a on smithi164 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4154002122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-mon[38106]: Deploying daemon grafana.a on smithi164 2024-03-26T05:22:48.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4154002122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:48 smithi164 ceph-mon[33554]: Deploying daemon grafana.a on smithi164 2024-03-26T05:22:48.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233999 got 231928234003 for osd.7 2024-03-26T05:22:48.725 DEBUG:teuthology.parallel:result is None 2024-03-26T05:22:49.052 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:48 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:48.781Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000462077s 2024-03-26T05:22:49.191 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:49.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:49 smithi164 ceph-mon[33554]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:49.749 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:49 smithi136 ceph-mon[29560]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:49.749 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:49 smithi136 ceph-mon[38106]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:49.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.000+0000 7fe63fd01700 1 -- 172.21.15.136:0/2982839781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe638100c20 msgr2=0x7fe638101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:49.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.000+0000 7fe63fd01700 1 --2- 172.21.15.136:0/2982839781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe638100c20 0x7fe638101000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe6340098f0 tx=0x7fe634009c00 comp rx=0 tx=0).stop 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.000+0000 7fe63fd01700 1 -- 172.21.15.136:0/2982839781 shutdown_connections 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.000+0000 7fe63fd01700 1 --2- 172.21.15.136:0/2982839781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638107440 0x7fe63810efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.000+0000 7fe63fd01700 1 --2- 172.21.15.136:0/2982839781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe638100c20 0x7fe638101000 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.000+0000 7fe63fd01700 1 --2- 172.21.15.136:0/2982839781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6381015d0 0x7fe638106d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.000+0000 7fe63fd01700 1 -- 172.21.15.136:0/2982839781 >> 172.21.15.136:0/2982839781 conn(0x7fe6380762c0 msgr2=0x7fe6380766d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 -- 172.21.15.136:0/2982839781 shutdown_connections 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 -- 172.21.15.136:0/2982839781 wait complete. 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 Processor -- start 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 -- start start 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638100c20 0x7fe638197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6381015d0 0x7fe638198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe638107440 0x7fe63819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63819cc10 con 0x7fe638100c20 2024-03-26T05:22:49.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63819cd80 con 0x7fe638107440 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63fd01700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63819cf40 con 0x7fe6381015d0 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63d29c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6381015d0 0x7fe638198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63d29c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6381015d0 0x7fe638198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:59156/0 (socket says 172.21.15.136:59156) 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.001+0000 7fe63da9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638100c20 0x7fe638197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63d29c700 1 -- 172.21.15.136:0/3246676131 learned_addr learned my addr 172.21.15.136:0/3246676131 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63e29e700 1 --2- 172.21.15.136:0/3246676131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe638107440 0x7fe63819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63d29c700 1 -- 172.21.15.136:0/3246676131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe638107440 msgr2=0x7fe63819c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63d29c700 1 --2- 172.21.15.136:0/3246676131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe638107440 0x7fe63819c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63d29c700 1 -- 172.21.15.136:0/3246676131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638100c20 msgr2=0x7fe638197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63d29c700 1 --2- 172.21.15.136:0/3246676131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638100c20 0x7fe638197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63d29c700 1 -- 172.21.15.136:0/3246676131 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe630009d00 con 0x7fe6381015d0 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63d29c700 1 --2- 172.21.15.136:0/3246676131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6381015d0 0x7fe638198270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe62800e940 tx=0x7fe62800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe62effd700 1 -- 172.21.15.136:0/3246676131 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628009730 con 0x7fe6381015d0 2024-03-26T05:22:49.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe62effd700 1 -- 172.21.15.136:0/3246676131 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe62800ce90 con 0x7fe6381015d0 2024-03-26T05:22:49.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe634009520 con 0x7fe6381015d0 2024-03-26T05:22:49.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe62effd700 1 -- 172.21.15.136:0/3246676131 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628018420 con 0x7fe6381015d0 2024-03-26T05:22:49.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.002+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6381a48b0 con 0x7fe6381015d0 2024-03-26T05:22:49.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.003+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe638068d40 con 0x7fe6381015d0 2024-03-26T05:22:49.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.004+0000 7fe62effd700 1 -- 172.21.15.136:0/3246676131 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7fe62800c9e0 con 0x7fe6381015d0 2024-03-26T05:22:49.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.004+0000 7fe62effd700 1 --2- 172.21.15.136:0/3246676131 >> v2:172.21.15.136:6800/3380174081 conn(0x7fe62406da80 0x7fe62406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:49.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.004+0000 7fe62effd700 1 -- 172.21.15.136:0/3246676131 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fe628014070 con 0x7fe6381015d0 2024-03-26T05:22:49.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.004+0000 7fe63da9d700 1 --2- 172.21.15.136:0/3246676131 >> v2:172.21.15.136:6800/3380174081 conn(0x7fe62406da80 0x7fe62406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:49.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.004+0000 7fe63da9d700 1 --2- 172.21.15.136:0/3246676131 >> v2:172.21.15.136:6800/3380174081 conn(0x7fe62406da80 0x7fe62406ff40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe634009fd0 tx=0x7fe634019040 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:49.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.009+0000 7fe62effd700 1 -- 172.21.15.136:0/3246676131 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe628053280 con 0x7fe6381015d0 2024-03-26T05:22:50.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.210+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe6381117f0 con 0x7fe6381015d0 2024-03-26T05:22:50.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.210+0000 7fe62effd700 1 -- 172.21.15.136:0/3246676131 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe62805a9c0 con 0x7fe6381015d0 2024-03-26T05:22:50.201 INFO:teuthology.orchestra.run.smithi136.stdout:34359738423 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 >> v2:172.21.15.136:6800/3380174081 conn(0x7fe62406da80 msgr2=0x7fe62406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 --2- 172.21.15.136:0/3246676131 >> v2:172.21.15.136:6800/3380174081 conn(0x7fe62406da80 0x7fe62406ff40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe634009fd0 tx=0x7fe634019040 comp rx=0 tx=0).stop 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6381015d0 msgr2=0x7fe638198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 --2- 172.21.15.136:0/3246676131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6381015d0 0x7fe638198270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe62800e940 tx=0x7fe62800ec50 comp rx=0 tx=0).stop 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 shutdown_connections 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 --2- 172.21.15.136:0/3246676131 >> v2:172.21.15.136:6800/3380174081 conn(0x7fe62406da80 0x7fe62406ff40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 --2- 172.21.15.136:0/3246676131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe638107440 0x7fe63819c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 --2- 172.21.15.136:0/3246676131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638100c20 0x7fe638197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 --2- 172.21.15.136:0/3246676131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6381015d0 0x7fe638198270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.212+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 >> 172.21.15.136:0/3246676131 conn(0x7fe6380762c0 msgr2=0x7fe6380fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.213+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 shutdown_connections 2024-03-26T05:22:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:50.213+0000 7fe63fd01700 1 -- 172.21.15.136:0/3246676131 wait complete. 2024-03-26T05:22:51.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:50 smithi136 ceph-mon[29560]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:51.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:50 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:51.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3246676131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:22:51.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:50 smithi136 ceph-mon[38106]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:51.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:50 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:51.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3246676131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:22:51.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757274 got 163208757278 for osd.5 2024-03-26T05:22:51.137 DEBUG:teuthology.parallel:result is None 2024-03-26T05:22:51.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:50 smithi164 ceph-mon[33554]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:51.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:50 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:22:51.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3246676131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T05:22:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/1052957098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98100f70 msgr2=0x7ffa981013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/1052957098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98100f70 0x7ffa981013f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009990 tx=0x7ffa8c009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/1052957098 shutdown_connections 2024-03-26T05:22:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/1052957098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa98106fb0 0x7ffa98107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/1052957098 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa98101930 0x7ffa9810ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/1052957098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98100f70 0x7ffa981013f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/1052957098 >> 172.21.15.136:0/1052957098 conn(0x7ffa98077f20 msgr2=0x7ffa98078330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:51.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/1052957098 shutdown_connections 2024-03-26T05:22:51.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.344+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/1052957098 wait complete. 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9cdf3700 1 Processor -- start 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9cdf3700 1 -- start start 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9cdf3700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa98100f70 0x7ffa9810e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9cdf3700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa98101930 0x7ffa9810ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9cdf3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98106fb0 0x7ffa9810f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9cdf3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981a6140 con 0x7ffa98106fb0 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9cdf3700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981a62b0 con 0x7ffa98101930 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9cdf3700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981a6470 con 0x7ffa98100f70 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9659c700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa98100f70 0x7ffa9810e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa95d9b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa98101930 0x7ffa9810ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa96d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98106fb0 0x7ffa9810f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:51.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa95d9b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa98101930 0x7ffa9810ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.136:58030/0 (socket says 172.21.15.136:58030) 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa95d9b700 1 -- 172.21.15.136:0/3202734100 learned_addr learned my addr 172.21.15.136:0/3202734100 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa95d9b700 1 -- 172.21.15.136:0/3202734100 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa98100f70 msgr2=0x7ffa9810e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa95d9b700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa98100f70 0x7ffa9810e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa95d9b700 1 -- 172.21.15.136:0/3202734100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98106fb0 msgr2=0x7ffa9810f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa95d9b700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98106fb0 0x7ffa9810f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa95d9b700 1 -- 172.21.15.136:0/3202734100 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90009c40 con 0x7ffa98101930 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa96d9d700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98106fb0 0x7ffa9810f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.345+0000 7ffa9659c700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa98100f70 0x7ffa9810e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.346+0000 7ffa95d9b700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa98101930 0x7ffa9810ed00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c000c00 tx=0x7ffa8c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.346+0000 7ffa877fe700 1 -- 172.21.15.136:0/3202734100 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c01d070 con 0x7ffa98101930 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.346+0000 7ffa877fe700 1 -- 172.21.15.136:0/3202734100 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ffa8c022470 con 0x7ffa98101930 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.346+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8c0095f0 con 0x7ffa98101930 2024-03-26T05:22:51.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.346+0000 7ffa877fe700 1 -- 172.21.15.136:0/3202734100 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c017650 con 0x7ffa98101930 2024-03-26T05:22:51.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.346+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa981a69d0 con 0x7ffa98101930 2024-03-26T05:22:51.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.347+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa981094d0 con 0x7ffa98101930 2024-03-26T05:22:51.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.347+0000 7ffa877fe700 1 -- 172.21.15.136:0/3202734100 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7ffa8c00fa90 con 0x7ffa98101930 2024-03-26T05:22:51.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.348+0000 7ffa877fe700 1 --2- 172.21.15.136:0/3202734100 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffa8006da80 0x7ffa8006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:51.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.348+0000 7ffa877fe700 1 -- 172.21.15.136:0/3202734100 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7ffa8c08e680 con 0x7ffa98101930 2024-03-26T05:22:51.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.348+0000 7ffa9659c700 1 --2- 172.21.15.136:0/3202734100 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffa8006da80 0x7ffa8006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:51.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.348+0000 7ffa9659c700 1 --2- 172.21.15.136:0/3202734100 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffa8006da80 0x7ffa8006ff40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009fd0 tx=0x7ffa88009260 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:51.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.352+0000 7ffa877fe700 1 -- 172.21.15.136:0/3202734100 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffa8c05bb00 con 0x7ffa98101930 2024-03-26T05:22:51.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.554+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ffa9810fcf0 con 0x7ffa98101930 2024-03-26T05:22:51.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.554+0000 7ffa877fe700 1 -- 172.21.15.136:0/3202734100 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffa8c05b690 con 0x7ffa98101930 2024-03-26T05:22:51.545 INFO:teuthology.orchestra.run.smithi136.stdout:55834574898 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffa8006da80 msgr2=0x7ffa8006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/3202734100 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffa8006da80 0x7ffa8006ff40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009fd0 tx=0x7ffa88009260 comp rx=0 tx=0).stop 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa98101930 msgr2=0x7ffa9810ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa98101930 0x7ffa9810ed00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c000c00 tx=0x7ffa8c004730 comp rx=0 tx=0).stop 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 shutdown_connections 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/3202734100 >> v2:172.21.15.136:6800/3380174081 conn(0x7ffa8006da80 0x7ffa8006ff40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa98106fb0 0x7ffa9810f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa98100f70 0x7ffa9810e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 --2- 172.21.15.136:0/3202734100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa98101930 0x7ffa9810ed00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 >> 172.21.15.136:0/3202734100 conn(0x7ffa98077f20 msgr2=0x7ffa980ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 shutdown_connections 2024-03-26T05:22:51.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:51.557+0000 7ffa9cdf3700 1 -- 172.21.15.136:0/3202734100 wait complete. 2024-03-26T05:22:52.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3202734100' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:22:52.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:22:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:22:52.291 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3202734100' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:22:52.292 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3202734100' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T05:22:52.325 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574895 got 55834574898 for osd.1 2024-03-26T05:22:52.326 DEBUG:teuthology.parallel:result is None 2024-03-26T05:22:53.155 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:52 smithi136 ceph-mon[38106]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:53.155 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:52 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:53.155 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:52 smithi136 ceph-mon[29560]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:53.155 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:52 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:53.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:52 smithi164 ceph-mon[33554]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:53.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:52 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:22:53.642 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738423 for osd.0 2024-03-26T05:22:53.642 DEBUG:teuthology.parallel:result is None 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.212+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/3374898078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101210 msgr2=0x7fbf341015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.212+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/3374898078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101210 0x7fbf341015f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbf300098f0 tx=0x7fbf30009c00 comp rx=0 tx=0).stop 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.212+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/3374898078 shutdown_connections 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.212+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/3374898078 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf341062a0 0x7fbf3410de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.212+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/3374898078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101210 0x7fbf341015f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.212+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/3374898078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf34101bc0 0x7fbf34105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.212+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/3374898078 >> 172.21.15.136:0/3374898078 conn(0x7fbf34076fa0 msgr2=0x7fbf340773b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/3374898078 shutdown_connections 2024-03-26T05:22:54.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/3374898078 wait complete. 2024-03-26T05:22:54.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 Processor -- start 2024-03-26T05:22:54.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 -- start start 2024-03-26T05:22:54.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf34101210 0x7fbf3419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:54.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101bc0 0x7fbf3419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf341062a0 0x7fbf34197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf341982e0 con 0x7fbf34101bc0 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf34198450 con 0x7fbf34101210 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3bfdc700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf34198610 con 0x7fbf341062a0 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf39577700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101bc0 0x7fbf3419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf3a579700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf341062a0 0x7fbf34197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf39577700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101bc0 0x7fbf3419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:57082/0 (socket says 172.21.15.136:57082) 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf39577700 1 -- 172.21.15.136:0/624767070 learned_addr learned my addr 172.21.15.136:0/624767070 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.213+0000 7fbf39d78700 1 --2- 172.21.15.136:0/624767070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf34101210 0x7fbf3419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf39577700 1 -- 172.21.15.136:0/624767070 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf341062a0 msgr2=0x7fbf34197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf39577700 1 --2- 172.21.15.136:0/624767070 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf341062a0 0x7fbf34197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf39577700 1 -- 172.21.15.136:0/624767070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf34101210 msgr2=0x7fbf3419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf39577700 1 --2- 172.21.15.136:0/624767070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf34101210 0x7fbf3419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf39577700 1 -- 172.21.15.136:0/624767070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c009d00 con 0x7fbf34101bc0 2024-03-26T05:22:54.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf39577700 1 --2- 172.21.15.136:0/624767070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101bc0 0x7fbf3419dfa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400d950 tx=0x7fbf2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:54.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf2affd700 1 -- 172.21.15.136:0/624767070 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24004220 con 0x7fbf34101bc0 2024-03-26T05:22:54.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf2affd700 1 -- 172.21.15.136:0/624767070 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fbf24007620 con 0x7fbf34101bc0 2024-03-26T05:22:54.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf2affd700 1 -- 172.21.15.136:0/624767070 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf240055e0 con 0x7fbf34101bc0 2024-03-26T05:22:54.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf30009520 con 0x7fbf34101bc0 2024-03-26T05:22:54.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.214+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf341a49e0 con 0x7fbf34101bc0 2024-03-26T05:22:54.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.215+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf34109530 con 0x7fbf34101bc0 2024-03-26T05:22:54.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.216+0000 7fbf2affd700 1 -- 172.21.15.136:0/624767070 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7fbf24004750 con 0x7fbf34101bc0 2024-03-26T05:22:54.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.216+0000 7fbf2affd700 1 --2- 172.21.15.136:0/624767070 >> v2:172.21.15.136:6800/3380174081 conn(0x7fbf2006da80 0x7fbf2006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:54.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.216+0000 7fbf2affd700 1 -- 172.21.15.136:0/624767070 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fbf2408d570 con 0x7fbf34101bc0 2024-03-26T05:22:54.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.216+0000 7fbf39d78700 1 --2- 172.21.15.136:0/624767070 >> v2:172.21.15.136:6800/3380174081 conn(0x7fbf2006da80 0x7fbf2006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:54.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.217+0000 7fbf39d78700 1 --2- 172.21.15.136:0/624767070 >> v2:172.21.15.136:6800/3380174081 conn(0x7fbf2006da80 0x7fbf2006ff40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009fd0 tx=0x7fbf30019040 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:54.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.220+0000 7fbf2affd700 1 -- 172.21.15.136:0/624767070 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbf2405aaf0 con 0x7fbf34101bc0 2024-03-26T05:22:54.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.416+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbf34198ff0 con 0x7fbf34101bc0 2024-03-26T05:22:54.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.417+0000 7fbf2affd700 1 -- 172.21.15.136:0/624767070 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbf2405a680 con 0x7fbf34101bc0 2024-03-26T05:22:54.407 INFO:teuthology.orchestra.run.smithi136.stdout:107374182441 2024-03-26T05:22:54.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 >> v2:172.21.15.136:6800/3380174081 conn(0x7fbf2006da80 msgr2=0x7fbf2006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:54.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/624767070 >> v2:172.21.15.136:6800/3380174081 conn(0x7fbf2006da80 0x7fbf2006ff40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009fd0 tx=0x7fbf30019040 comp rx=0 tx=0).stop 2024-03-26T05:22:54.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101bc0 msgr2=0x7fbf3419dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:54.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/624767070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101bc0 0x7fbf3419dfa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400d950 tx=0x7fbf2400dc60 comp rx=0 tx=0).stop 2024-03-26T05:22:54.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 shutdown_connections 2024-03-26T05:22:54.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/624767070 >> v2:172.21.15.136:6800/3380174081 conn(0x7fbf2006da80 0x7fbf2006ff40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/624767070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf34101210 0x7fbf3419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/624767070 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf341062a0 0x7fbf34197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 --2- 172.21.15.136:0/624767070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf34101bc0 0x7fbf3419dfa0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:54.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 >> 172.21.15.136:0/624767070 conn(0x7fbf34076fa0 msgr2=0x7fbf34107420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:54.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 shutdown_connections 2024-03-26T05:22:54.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:54.419+0000 7fbf3bfdc700 1 -- 172.21.15.136:0/624767070 wait complete. 2024-03-26T05:22:55.066 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:54 smithi136 ceph-mon[38106]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:55.066 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/624767070' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:22:55.066 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:54 smithi136 ceph-mon[29560]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:55.066 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/624767070' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:22:55.095 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182437 got 107374182441 for osd.3 2024-03-26T05:22:55.095 DEBUG:teuthology.parallel:result is None 2024-03-26T05:22:55.095 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T05:22:55.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:22:55.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:54 smithi164 ceph-mon[33554]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:55.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/624767070' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T05:22:55.337 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:56.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:55 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:56.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:55 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:56.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:55 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:22:57.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:56 smithi136 ceph-mon[38106]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:57.038 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:56 smithi136 ceph-mon[29560]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:57.038 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:22:56 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:22:56.784Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003356753s 2024-03-26T05:22:57.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 -- 172.21.15.136:0/4213993270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44100f80 msgr2=0x7f2d441013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:57.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 --2- 172.21.15.136:0/4213993270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44100f80 0x7f2d441013e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009990 tx=0x7f2d34009ca0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 -- 172.21.15.136:0/4213993270 shutdown_connections 2024-03-26T05:22:57.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 --2- 172.21.15.136:0/4213993270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d44106fc0 0x7f2d441073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 --2- 172.21.15.136:0/4213993270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d44101920 0x7f2d4410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 --2- 172.21.15.136:0/4213993270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44100f80 0x7f2d441013e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 -- 172.21.15.136:0/4213993270 >> 172.21.15.136:0/4213993270 conn(0x7f2d44077eb0 msgr2=0x7f2d440782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 -- 172.21.15.136:0/4213993270 shutdown_connections 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 -- 172.21.15.136:0/4213993270 wait complete. 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 Processor -- start 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.155+0000 7f2d4c710700 1 -- start start 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4c710700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d44100f80 0x7f2d441969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4c710700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d44101920 0x7f2d44196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4c710700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44106fc0 0x7f2d44197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4c710700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d441a60c0 con 0x7f2d44100f80 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4c710700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d441a6200 con 0x7f2d44101920 2024-03-26T05:22:57.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4c710700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d441a63c0 con 0x7f2d44106fc0 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44106fc0 0x7f2d44197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4a4ac700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d44100f80 0x7f2d441969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44106fc0 0x7f2d44197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:57746/0 (socket says 172.21.15.136:57746) 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 -- 172.21.15.136:0/3041623881 learned_addr learned my addr 172.21.15.136:0/3041623881 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d49cab700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d44101920 0x7f2d44196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 -- 172.21.15.136:0/3041623881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d44101920 msgr2=0x7f2d44196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d44101920 0x7f2d44196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 -- 172.21.15.136:0/3041623881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d44100f80 msgr2=0x7f2d441969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d44100f80 0x7f2d441969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 -- 172.21.15.136:0/3041623881 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c009c40 con 0x7f2d44106fc0 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d49cab700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d44101920 0x7f2d44196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4a4ac700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d44100f80 0x7f2d441969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.156+0000 7f2d4acad700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44106fc0 0x7f2d44197430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00c3a0 tx=0x7f2d3c00c6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:57.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.157+0000 7f2d3b7fe700 1 -- 172.21.15.136:0/3041623881 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c017070 con 0x7f2d44106fc0 2024-03-26T05:22:57.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.157+0000 7f2d3b7fe700 1 -- 172.21.15.136:0/3041623881 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f2d3c00f040 con 0x7f2d44106fc0 2024-03-26T05:22:57.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.157+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d340095f0 con 0x7f2d44106fc0 2024-03-26T05:22:57.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.157+0000 7f2d3b7fe700 1 -- 172.21.15.136:0/3041623881 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c00e050 con 0x7f2d44106fc0 2024-03-26T05:22:57.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.157+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d441a69b0 con 0x7f2d44106fc0 2024-03-26T05:22:57.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.158+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d441094e0 con 0x7f2d44106fc0 2024-03-26T05:22:57.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.158+0000 7f2d3b7fe700 1 -- 172.21.15.136:0/3041623881 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f2d3c00cb90 con 0x7f2d44106fc0 2024-03-26T05:22:57.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.159+0000 7f2d3b7fe700 1 --2- 172.21.15.136:0/3041623881 >> v2:172.21.15.136:6800/3380174081 conn(0x7f2d3006da80 0x7f2d3006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:22:57.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.159+0000 7f2d4a4ac700 1 --2- 172.21.15.136:0/3041623881 >> v2:172.21.15.136:6800/3380174081 conn(0x7f2d3006da80 0x7f2d3006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:22:57.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.159+0000 7f2d3b7fe700 1 -- 172.21.15.136:0/3041623881 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f2d3c08ccc0 con 0x7f2d44106fc0 2024-03-26T05:22:57.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.159+0000 7f2d4a4ac700 1 --2- 172.21.15.136:0/3041623881 >> v2:172.21.15.136:6800/3380174081 conn(0x7f2d3006da80 0x7f2d3006ff40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009fd0 tx=0x7f2d40009260 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:22:57.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.163+0000 7f2d3b7fe700 1 -- 172.21.15.136:0/3041623881 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2d3c05a160 con 0x7f2d44106fc0 2024-03-26T05:22:57.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:56 smithi164 ceph-mon[33554]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:22:57.324 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:22:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:22:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:22:57.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.383+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 --> v2:172.21.15.136:6800/3380174081 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d4406a2d0 con 0x7f2d3006da80 2024-03-26T05:22:57.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.402+0000 7f2d3b7fe700 1 -- 172.21.15.136:0/3041623881 <== mgr.24485 v2:172.21.15.136:6800/3380174081 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346510 (secure 0 0 0) 0x7f2d4406a2d0 con 0x7f2d3006da80 2024-03-26T05:22:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 >> v2:172.21.15.136:6800/3380174081 conn(0x7f2d3006da80 msgr2=0x7f2d3006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 --2- 172.21.15.136:0/3041623881 >> v2:172.21.15.136:6800/3380174081 conn(0x7f2d3006da80 0x7f2d3006ff40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009fd0 tx=0x7f2d40009260 comp rx=0 tx=0).stop 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44106fc0 msgr2=0x7f2d44197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44106fc0 0x7f2d44197430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00c3a0 tx=0x7f2d3c00c6b0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 shutdown_connections 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 --2- 172.21.15.136:0/3041623881 >> v2:172.21.15.136:6800/3380174081 conn(0x7f2d3006da80 0x7f2d3006ff40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d44106fc0 0x7f2d44197430 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d44100f80 0x7f2d441969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 --2- 172.21.15.136:0/3041623881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d44101920 0x7f2d44196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 >> 172.21.15.136:0/3041623881 conn(0x7f2d44077eb0 msgr2=0x7f2d440ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 shutdown_connections 2024-03-26T05:22:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:22:57.407+0000 7f2d4c710700 1 -- 172.21.15.136:0/3041623881 wait complete. 2024-03-26T05:22:57.402 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-03-26T05:22:58.043 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:57 smithi136 ceph-mon[38106]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:58.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:57 smithi136 ceph-mon[29560]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:58.075 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-26T05:22:55.828071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1301,"num_read_kb":1115,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":796,"ondisk_log_size":796,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240180,"kb_used_data":7388,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721036,"statfs":{"total":767960285184,"available":767714340864,"internally_reserved":0,"allocated":7565312,"data_stored":4027914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":238276147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003077"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808748+0000","last_change":"2024-03-26T05:21:36.855842+0000","last_active":"2024-03-26T05:22:09.808748+0000","last_peered":"2024-03-26T05:22:09.808748+0000","last_clean":"2024-03-26T05:22:09.808748+0000","last_became_active":"2024-03-26T05:21:36.855239+0000","last_became_peered":"2024-03-26T05:21:36.855239+0000","last_unstale":"2024-03-26T05:22:09.808748+0000","last_undegraded":"2024-03-26T05:22:09.808748+0000","last_fullsized":"2024-03-26T05:22:09.808748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:42:47.079545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990571+0000","last_change":"2024-03-26T05:21:34.849980+0000","last_active":"2024-03-26T05:22:09.990571+0000","last_peered":"2024-03-26T05:22:09.990571+0000","last_clean":"2024-03-26T05:22:09.990571+0000","last_became_active":"2024-03-26T05:21:34.849655+0000","last_became_peered":"2024-03-26T05:21:34.849655+0000","last_unstale":"2024-03-26T05:22:09.990571+0000","last_undegraded":"2024-03-26T05:22:09.990571+0000","last_fullsized":"2024-03-26T05:22:09.990571+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:07:08.932699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992050+0000","last_change":"2024-03-26T05:21:30.831397+0000","last_active":"2024-03-26T05:22:09.992050+0000","last_peered":"2024-03-26T05:22:09.992050+0000","last_clean":"2024-03-26T05:22:09.992050+0000","last_became_active":"2024-03-26T05:21:30.831015+0000","last_became_peered":"2024-03-26T05:21:30.831015+0000","last_unstale":"2024-03-26T05:22:09.992050+0000","last_undegraded":"2024-03-26T05:22:09.992050+0000","last_fullsized":"2024-03-26T05:22:09.992050+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:58:57.861979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'16","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808921+0000","last_change":"2024-03-26T05:21:57.694124+0000","last_active":"2024-03-26T05:22:09.808921+0000","last_peered":"2024-03-26T05:22:09.808921+0000","last_clean":"2024-03-26T05:22:09.808921+0000","last_became_active":"2024-03-26T05:21:57.687312+0000","last_became_peered":"2024-03-26T05:21:57.687312+0000","last_unstale":"2024-03-26T05:22:09.808921+0000","last_undegraded":"2024-03-26T05:22:09.808921+0000","last_fullsized":"2024-03-26T05:22:09.808921+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:59:31.535805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807401+0000","last_change":"2024-03-26T05:21:37.149332+0000","last_active":"2024-03-26T05:22:09.807401+0000","last_peered":"2024-03-26T05:22:09.807401+0000","last_clean":"2024-03-26T05:22:09.807401+0000","last_became_active":"2024-03-26T05:21:37.149032+0000","last_became_peered":"2024-03-26T05:21:37.149032+0000","last_unstale":"2024-03-26T05:22:09.807401+0000","last_undegraded":"2024-03-26T05:22:09.807401+0000","last_fullsized":"2024-03-26T05:22:09.807401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:12:11.259814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808750+0000","last_change":"2024-03-26T05:21:34.847322+0000","last_active":"2024-03-26T05:22:09.808750+0000","last_peered":"2024-03-26T05:22:09.808750+0000","last_clean":"2024-03-26T05:22:09.808750+0000","last_became_active":"2024-03-26T05:21:34.847024+0000","last_became_peered":"2024-03-26T05:21:34.847024+0000","last_unstale":"2024-03-26T05:22:09.808750+0000","last_undegraded":"2024-03-26T05:22:09.808750+0000","last_fullsized":"2024-03-26T05:22:09.808750+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:00:10.781213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809505+0000","last_change":"2024-03-26T05:21:30.824036+0000","last_active":"2024-03-26T05:22:09.809505+0000","last_peered":"2024-03-26T05:22:09.809505+0000","last_clean":"2024-03-26T05:22:09.809505+0000","last_became_active":"2024-03-26T05:21:30.823731+0000","last_became_peered":"2024-03-26T05:21:30.823731+0000","last_unstale":"2024-03-26T05:22:09.809505+0000","last_undegraded":"2024-03-26T05:22:09.809505+0000","last_fullsized":"2024-03-26T05:22:09.809505+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:43:01.615281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'39","reported_seq":102,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806953+0000","last_change":"2024-03-26T05:21:32.847218+0000","last_active":"2024-03-26T05:22:09.806953+0000","last_peered":"2024-03-26T05:22:09.806953+0000","last_clean":"2024-03-26T05:22:09.806953+0000","last_became_active":"2024-03-26T05:21:32.846590+0000","last_became_peered":"2024-03-26T05:21:32.846590+0000","last_unstale":"2024-03-26T05:22:09.806953+0000","last_undegraded":"2024-03-26T05:22:09.806953+0000","last_fullsized":"2024-03-26T05:22:09.806953+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:49:18.147095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991232+0000","last_change":"2024-03-26T05:21:36.854407+0000","last_active":"2024-03-26T05:22:09.991232+0000","last_peered":"2024-03-26T05:22:09.991232+0000","last_clean":"2024-03-26T05:22:09.991232+0000","last_became_active":"2024-03-26T05:21:36.854053+0000","last_became_peered":"2024-03-26T05:21:36.854053+0000","last_unstale":"2024-03-26T05:22:09.991232+0000","last_undegraded":"2024-03-26T05:22:09.991232+0000","last_fullsized":"2024-03-26T05:22:09.991232+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:06:31.584429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911883+0000","last_change":"2024-03-26T05:21:34.853554+0000","last_active":"2024-03-26T05:22:09.911883+0000","last_peered":"2024-03-26T05:22:09.911883+0000","last_clean":"2024-03-26T05:22:09.911883+0000","last_became_active":"2024-03-26T05:21:34.853242+0000","last_became_peered":"2024-03-26T05:21:34.853242+0000","last_unstale":"2024-03-26T05:22:09.911883+0000","last_undegraded":"2024-03-26T05:22:09.911883+0000","last_fullsized":"2024-03-26T05:22:09.911883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:15:16.686440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911970+0000","last_change":"2024-03-26T05:21:30.831889+0000","last_active":"2024-03-26T05:22:09.911970+0000","last_peered":"2024-03-26T05:22:09.911970+0000","last_clean":"2024-03-26T05:22:09.911970+0000","last_became_active":"2024-03-26T05:21:30.830857+0000","last_became_peered":"2024-03-26T05:21:30.830857+0000","last_unstale":"2024-03-26T05:22:09.911970+0000","last_undegraded":"2024-03-26T05:22:09.911970+0000","last_fullsized":"2024-03-26T05:22:09.911970+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:41:15.266182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'21","reported_seq":69,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810847+0000","last_change":"2024-03-26T05:21:32.842463+0000","last_active":"2024-03-26T05:22:09.810847+0000","last_peered":"2024-03-26T05:22:09.810847+0000","last_clean":"2024-03-26T05:22:09.810847+0000","last_became_active":"2024-03-26T05:21:32.842119+0000","last_became_peered":"2024-03-26T05:21:32.842119+0000","last_unstale":"2024-03-26T05:22:09.810847+0000","last_undegraded":"2024-03-26T05:22:09.810847+0000","last_fullsized":"2024-03-26T05:22:09.810847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:24.029117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911566+0000","last_change":"2024-03-26T05:21:36.855058+0000","last_active":"2024-03-26T05:22:09.911566+0000","last_peered":"2024-03-26T05:22:09.911566+0000","last_clean":"2024-03-26T05:22:09.911566+0000","last_became_active":"2024-03-26T05:21:36.854745+0000","last_became_peered":"2024-03-26T05:21:36.854745+0000","last_unstale":"2024-03-26T05:22:09.911566+0000","last_undegraded":"2024-03-26T05:22:09.911566+0000","last_fullsized":"2024-03-26T05:22:09.911566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:53:04.473004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923015+0000","last_change":"2024-03-26T05:21:34.845990+0000","last_active":"2024-03-26T05:22:09.923015+0000","last_peered":"2024-03-26T05:22:09.923015+0000","last_clean":"2024-03-26T05:22:09.923015+0000","last_became_active":"2024-03-26T05:21:34.845529+0000","last_became_peered":"2024-03-26T05:21:34.845529+0000","last_unstale":"2024-03-26T05:22:09.923015+0000","last_undegraded":"2024-03-26T05:22:09.923015+0000","last_fullsized":"2024-03-26T05:22:09.923015+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:59:10.295747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911916+0000","last_change":"2024-03-26T05:21:30.831347+0000","last_active":"2024-03-26T05:22:09.911916+0000","last_peered":"2024-03-26T05:22:09.911916+0000","last_clean":"2024-03-26T05:22:09.911916+0000","last_became_active":"2024-03-26T05:21:30.831039+0000","last_became_peered":"2024-03-26T05:21:30.831039+0000","last_unstale":"2024-03-26T05:22:09.911916+0000","last_undegraded":"2024-03-26T05:22:09.911916+0000","last_fullsized":"2024-03-26T05:22:09.911916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:40:05.305903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'11","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809301+0000","last_change":"2024-03-26T05:21:32.849756+0000","last_active":"2024-03-26T05:22:09.809301+0000","last_peered":"2024-03-26T05:22:09.809301+0000","last_clean":"2024-03-26T05:22:09.809301+0000","last_became_active":"2024-03-26T05:21:32.848807+0000","last_became_peered":"2024-03-26T05:21:32.848807+0000","last_unstale":"2024-03-26T05:22:09.809301+0000","last_undegraded":"2024-03-26T05:22:09.809301+0000","last_fullsized":"2024-03-26T05:22:09.809301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:52:55.525821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809733+0000","last_change":"2024-03-26T05:21:37.148267+0000","last_active":"2024-03-26T05:22:09.809733+0000","last_peered":"2024-03-26T05:22:09.809733+0000","last_clean":"2024-03-26T05:22:09.809733+0000","last_became_active":"2024-03-26T05:21:37.147971+0000","last_became_peered":"2024-03-26T05:21:37.147971+0000","last_unstale":"2024-03-26T05:22:09.809733+0000","last_undegraded":"2024-03-26T05:22:09.809733+0000","last_fullsized":"2024-03-26T05:22:09.809733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:03:32.847453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991521+0000","last_change":"2024-03-26T05:21:34.854477+0000","last_active":"2024-03-26T05:22:09.991521+0000","last_peered":"2024-03-26T05:22:09.991521+0000","last_clean":"2024-03-26T05:22:09.991521+0000","last_became_active":"2024-03-26T05:21:34.854052+0000","last_became_peered":"2024-03-26T05:21:34.854052+0000","last_unstale":"2024-03-26T05:22:09.991521+0000","last_undegraded":"2024-03-26T05:22:09.991521+0000","last_fullsized":"2024-03-26T05:22:09.991521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:56:19.159091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992646+0000","last_change":"2024-03-26T05:21:30.833534+0000","last_active":"2024-03-26T05:22:09.992646+0000","last_peered":"2024-03-26T05:22:09.992646+0000","last_clean":"2024-03-26T05:22:09.992646+0000","last_became_active":"2024-03-26T05:21:30.832166+0000","last_became_peered":"2024-03-26T05:21:30.832166+0000","last_unstale":"2024-03-26T05:22:09.992646+0000","last_undegraded":"2024-03-26T05:22:09.992646+0000","last_fullsized":"2024-03-26T05:22:09.992646+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:33:34.525001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807220+0000","last_change":"2024-03-26T05:21:57.805553+0000","last_active":"2024-03-26T05:22:09.807220+0000","last_peered":"2024-03-26T05:22:09.807220+0000","last_clean":"2024-03-26T05:22:09.807220+0000","last_became_active":"2024-03-26T05:21:57.796393+0000","last_became_peered":"2024-03-26T05:21:57.796393+0000","last_unstale":"2024-03-26T05:22:09.807220+0000","last_undegraded":"2024-03-26T05:22:09.807220+0000","last_fullsized":"2024-03-26T05:22:09.807220+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:43:24.576405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808727+0000","last_change":"2024-03-26T05:21:37.148629+0000","last_active":"2024-03-26T05:22:09.808727+0000","last_peered":"2024-03-26T05:22:09.808727+0000","last_clean":"2024-03-26T05:22:09.808727+0000","last_became_active":"2024-03-26T05:21:37.148268+0000","last_became_peered":"2024-03-26T05:21:37.148268+0000","last_unstale":"2024-03-26T05:22:09.808727+0000","last_undegraded":"2024-03-26T05:22:09.808727+0000","last_fullsized":"2024-03-26T05:22:09.808727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:25:35.698464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809582+0000","last_change":"2024-03-26T05:21:34.842701+0000","last_active":"2024-03-26T05:22:09.809582+0000","last_peered":"2024-03-26T05:22:09.809582+0000","last_clean":"2024-03-26T05:22:09.809582+0000","last_became_active":"2024-03-26T05:21:34.842427+0000","last_became_peered":"2024-03-26T05:21:34.842427+0000","last_unstale":"2024-03-26T05:22:09.809582+0000","last_undegraded":"2024-03-26T05:22:09.809582+0000","last_fullsized":"2024-03-26T05:22:09.809582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:53:25.426656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810982+0000","last_change":"2024-03-26T05:21:30.832724+0000","last_active":"2024-03-26T05:22:09.810982+0000","last_peered":"2024-03-26T05:22:09.810982+0000","last_clean":"2024-03-26T05:22:09.810982+0000","last_became_active":"2024-03-26T05:21:30.832143+0000","last_became_peered":"2024-03-26T05:21:30.832143+0000","last_unstale":"2024-03-26T05:22:09.810982+0000","last_undegraded":"2024-03-26T05:22:09.810982+0000","last_fullsized":"2024-03-26T05:22:09.810982+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:35:24.369382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'18","reported_seq":64,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807603+0000","last_change":"2024-03-26T05:21:32.850276+0000","last_active":"2024-03-26T05:22:09.807603+0000","last_peered":"2024-03-26T05:22:09.807603+0000","last_clean":"2024-03-26T05:22:09.807603+0000","last_became_active":"2024-03-26T05:21:32.849944+0000","last_became_peered":"2024-03-26T05:21:32.849944+0000","last_unstale":"2024-03-26T05:22:09.807603+0000","last_undegraded":"2024-03-26T05:22:09.807603+0000","last_fullsized":"2024-03-26T05:22:09.807603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:03:43.659584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923372+0000","last_change":"2024-03-26T05:21:36.857667+0000","last_active":"2024-03-26T05:22:09.923372+0000","last_peered":"2024-03-26T05:22:09.923372+0000","last_clean":"2024-03-26T05:22:09.923372+0000","last_became_active":"2024-03-26T05:21:36.857364+0000","last_became_peered":"2024-03-26T05:21:36.857364+0000","last_unstale":"2024-03-26T05:22:09.923372+0000","last_undegraded":"2024-03-26T05:22:09.923372+0000","last_fullsized":"2024-03-26T05:22:09.923372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:39:00.764868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889827+0000","last_change":"2024-03-26T05:21:34.850668+0000","last_active":"2024-03-26T05:22:45.889827+0000","last_peered":"2024-03-26T05:22:45.889827+0000","last_clean":"2024-03-26T05:22:45.889827+0000","last_became_active":"2024-03-26T05:21:34.850210+0000","last_became_peered":"2024-03-26T05:21:34.850210+0000","last_unstale":"2024-03-26T05:22:45.889827+0000","last_undegraded":"2024-03-26T05:22:45.889827+0000","last_fullsized":"2024-03-26T05:22:45.889827+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:50:15.781237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'2","reported_seq":40,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990670+0000","last_change":"2024-03-26T05:21:30.825374+0000","last_active":"2024-03-26T05:22:09.990670+0000","last_peered":"2024-03-26T05:22:09.990670+0000","last_clean":"2024-03-26T05:22:09.990670+0000","last_became_active":"2024-03-26T05:21:30.825023+0000","last_became_peered":"2024-03-26T05:21:30.825023+0000","last_unstale":"2024-03-26T05:22:09.990670+0000","last_undegraded":"2024-03-26T05:22:09.990670+0000","last_fullsized":"2024-03-26T05:22:09.990670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:20:41.536229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'16","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992595+0000","last_change":"2024-03-26T05:21:33.318076+0000","last_active":"2024-03-26T05:22:09.992595+0000","last_peered":"2024-03-26T05:22:09.992595+0000","last_clean":"2024-03-26T05:22:09.992595+0000","last_became_active":"2024-03-26T05:21:33.317781+0000","last_became_peered":"2024-03-26T05:21:33.317781+0000","last_unstale":"2024-03-26T05:22:09.992595+0000","last_undegraded":"2024-03-26T05:22:09.992595+0000","last_fullsized":"2024-03-26T05:22:09.992595+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:04:28.183791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991268+0000","last_change":"2024-03-26T05:21:36.857211+0000","last_active":"2024-03-26T05:22:09.991268+0000","last_peered":"2024-03-26T05:22:09.991268+0000","last_clean":"2024-03-26T05:22:09.991268+0000","last_became_active":"2024-03-26T05:21:36.856677+0000","last_became_peered":"2024-03-26T05:21:36.856677+0000","last_unstale":"2024-03-26T05:22:09.991268+0000","last_undegraded":"2024-03-26T05:22:09.991268+0000","last_fullsized":"2024-03-26T05:22:09.991268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:19:12.371760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992920+0000","last_change":"2024-03-26T05:21:34.855607+0000","last_active":"2024-03-26T05:22:09.992920+0000","last_peered":"2024-03-26T05:22:09.992920+0000","last_clean":"2024-03-26T05:22:09.992920+0000","last_became_active":"2024-03-26T05:21:34.855231+0000","last_became_peered":"2024-03-26T05:21:34.855231+0000","last_unstale":"2024-03-26T05:22:09.992920+0000","last_undegraded":"2024-03-26T05:22:09.992920+0000","last_fullsized":"2024-03-26T05:22:09.992920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:53.782244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807424+0000","last_change":"2024-03-26T05:21:30.826683+0000","last_active":"2024-03-26T05:22:09.807424+0000","last_peered":"2024-03-26T05:22:09.807424+0000","last_clean":"2024-03-26T05:22:09.807424+0000","last_became_active":"2024-03-26T05:21:30.826119+0000","last_became_peered":"2024-03-26T05:21:30.826119+0000","last_unstale":"2024-03-26T05:22:09.807424+0000","last_undegraded":"2024-03-26T05:22:09.807424+0000","last_fullsized":"2024-03-26T05:22:09.807424+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:42:31.524212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990761+0000","last_change":"2024-03-26T05:21:32.849949+0000","last_active":"2024-03-26T05:22:09.990761+0000","last_peered":"2024-03-26T05:22:09.990761+0000","last_clean":"2024-03-26T05:22:09.990761+0000","last_became_active":"2024-03-26T05:21:32.849625+0000","last_became_peered":"2024-03-26T05:21:32.849625+0000","last_unstale":"2024-03-26T05:22:09.990761+0000","last_undegraded":"2024-03-26T05:22:09.990761+0000","last_fullsized":"2024-03-26T05:22:09.990761+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:26:19.461625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923025+0000","last_change":"2024-03-26T05:21:36.857211+0000","last_active":"2024-03-26T05:22:09.923025+0000","last_peered":"2024-03-26T05:22:09.923025+0000","last_clean":"2024-03-26T05:22:09.923025+0000","last_became_active":"2024-03-26T05:21:36.856918+0000","last_became_peered":"2024-03-26T05:21:36.856918+0000","last_unstale":"2024-03-26T05:22:09.923025+0000","last_undegraded":"2024-03-26T05:22:09.923025+0000","last_fullsized":"2024-03-26T05:22:09.923025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:18:17.199617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990271+0000","last_change":"2024-03-26T05:21:34.848601+0000","last_active":"2024-03-26T05:22:09.990271+0000","last_peered":"2024-03-26T05:22:09.990271+0000","last_clean":"2024-03-26T05:22:09.990271+0000","last_became_active":"2024-03-26T05:21:34.848219+0000","last_became_peered":"2024-03-26T05:21:34.848219+0000","last_unstale":"2024-03-26T05:22:09.990271+0000","last_undegraded":"2024-03-26T05:22:09.990271+0000","last_fullsized":"2024-03-26T05:22:09.990271+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:19:07.038676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992141+0000","last_change":"2024-03-26T05:21:30.826495+0000","last_active":"2024-03-26T05:22:09.992141+0000","last_peered":"2024-03-26T05:22:09.992141+0000","last_clean":"2024-03-26T05:22:09.992141+0000","last_became_active":"2024-03-26T05:21:30.825954+0000","last_became_peered":"2024-03-26T05:21:30.825954+0000","last_unstale":"2024-03-26T05:22:09.992141+0000","last_undegraded":"2024-03-26T05:22:09.992141+0000","last_fullsized":"2024-03-26T05:22:09.992141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:32:24.921786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809973+0000","last_change":"2024-03-26T05:21:32.851635+0000","last_active":"2024-03-26T05:22:09.809973+0000","last_peered":"2024-03-26T05:22:09.809973+0000","last_clean":"2024-03-26T05:22:09.809973+0000","last_became_active":"2024-03-26T05:21:32.851323+0000","last_became_peered":"2024-03-26T05:21:32.851323+0000","last_unstale":"2024-03-26T05:22:09.809973+0000","last_undegraded":"2024-03-26T05:22:09.809973+0000","last_fullsized":"2024-03-26T05:22:09.809973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:05:50.958504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806352+0000","last_change":"2024-03-26T05:21:37.150189+0000","last_active":"2024-03-26T05:22:09.806352+0000","last_peered":"2024-03-26T05:22:09.806352+0000","last_clean":"2024-03-26T05:22:09.806352+0000","last_became_active":"2024-03-26T05:21:37.149777+0000","last_became_peered":"2024-03-26T05:21:37.149777+0000","last_unstale":"2024-03-26T05:22:09.806352+0000","last_undegraded":"2024-03-26T05:22:09.806352+0000","last_fullsized":"2024-03-26T05:22:09.806352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:10:40.698869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923196+0000","last_change":"2024-03-26T05:21:34.848587+0000","last_active":"2024-03-26T05:22:09.923196+0000","last_peered":"2024-03-26T05:22:09.923196+0000","last_clean":"2024-03-26T05:22:09.923196+0000","last_became_active":"2024-03-26T05:21:34.848279+0000","last_became_peered":"2024-03-26T05:21:34.848279+0000","last_unstale":"2024-03-26T05:22:09.923196+0000","last_undegraded":"2024-03-26T05:22:09.923196+0000","last_fullsized":"2024-03-26T05:22:09.923196+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:22:37.469760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912177+0000","last_change":"2024-03-26T05:21:30.824844+0000","last_active":"2024-03-26T05:22:09.912177+0000","last_peered":"2024-03-26T05:22:09.912177+0000","last_clean":"2024-03-26T05:22:09.912177+0000","last_became_active":"2024-03-26T05:21:30.824528+0000","last_became_peered":"2024-03-26T05:21:30.824528+0000","last_unstale":"2024-03-26T05:22:09.912177+0000","last_undegraded":"2024-03-26T05:22:09.912177+0000","last_fullsized":"2024-03-26T05:22:09.912177+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:34:03.013425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806738+0000","last_change":"2024-03-26T05:21:33.315420+0000","last_active":"2024-03-26T05:22:09.806738+0000","last_peered":"2024-03-26T05:22:09.806738+0000","last_clean":"2024-03-26T05:22:09.806738+0000","last_became_active":"2024-03-26T05:21:33.315099+0000","last_became_peered":"2024-03-26T05:21:33.315099+0000","last_unstale":"2024-03-26T05:22:09.806738+0000","last_undegraded":"2024-03-26T05:22:09.806738+0000","last_fullsized":"2024-03-26T05:22:09.806738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:04:06.283537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990334+0000","last_change":"2024-03-26T05:21:36.856760+0000","last_active":"2024-03-26T05:22:09.990334+0000","last_peered":"2024-03-26T05:22:09.990334+0000","last_clean":"2024-03-26T05:22:09.990334+0000","last_became_active":"2024-03-26T05:21:36.856457+0000","last_became_peered":"2024-03-26T05:21:36.856457+0000","last_unstale":"2024-03-26T05:22:09.990334+0000","last_undegraded":"2024-03-26T05:22:09.990334+0000","last_fullsized":"2024-03-26T05:22:09.990334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:56:41.932698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810368+0000","last_change":"2024-03-26T05:21:34.857757+0000","last_active":"2024-03-26T05:22:09.810368+0000","last_peered":"2024-03-26T05:22:09.810368+0000","last_clean":"2024-03-26T05:22:09.810368+0000","last_became_active":"2024-03-26T05:21:34.857396+0000","last_became_peered":"2024-03-26T05:21:34.857396+0000","last_unstale":"2024-03-26T05:22:09.810368+0000","last_undegraded":"2024-03-26T05:22:09.810368+0000","last_fullsized":"2024-03-26T05:22:09.810368+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:12.424403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923557+0000","last_change":"2024-03-26T05:21:30.833235+0000","last_active":"2024-03-26T05:22:09.923557+0000","last_peered":"2024-03-26T05:22:09.923557+0000","last_clean":"2024-03-26T05:22:09.923557+0000","last_became_active":"2024-03-26T05:21:30.832498+0000","last_became_peered":"2024-03-26T05:21:30.832498+0000","last_unstale":"2024-03-26T05:22:09.923557+0000","last_undegraded":"2024-03-26T05:22:09.923557+0000","last_fullsized":"2024-03-26T05:22:09.923557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:59:17.855155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991170+0000","last_change":"2024-03-26T05:21:32.847793+0000","last_active":"2024-03-26T05:22:09.991170+0000","last_peered":"2024-03-26T05:22:09.991170+0000","last_clean":"2024-03-26T05:22:09.991170+0000","last_became_active":"2024-03-26T05:21:32.847468+0000","last_became_peered":"2024-03-26T05:21:32.847468+0000","last_unstale":"2024-03-26T05:22:09.991170+0000","last_undegraded":"2024-03-26T05:22:09.991170+0000","last_fullsized":"2024-03-26T05:22:09.991170+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:26:47.840583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806414+0000","last_change":"2024-03-26T05:21:36.857092+0000","last_active":"2024-03-26T05:22:09.806414+0000","last_peered":"2024-03-26T05:22:09.806414+0000","last_clean":"2024-03-26T05:22:09.806414+0000","last_became_active":"2024-03-26T05:21:36.856471+0000","last_became_peered":"2024-03-26T05:21:36.856471+0000","last_unstale":"2024-03-26T05:22:09.806414+0000","last_undegraded":"2024-03-26T05:22:09.806414+0000","last_fullsized":"2024-03-26T05:22:09.806414+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:52:34.569101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990900+0000","last_change":"2024-03-26T05:21:34.850613+0000","last_active":"2024-03-26T05:22:09.990900+0000","last_peered":"2024-03-26T05:22:09.990900+0000","last_clean":"2024-03-26T05:22:09.990900+0000","last_became_active":"2024-03-26T05:21:34.849544+0000","last_became_peered":"2024-03-26T05:21:34.849544+0000","last_unstale":"2024-03-26T05:22:09.990900+0000","last_undegraded":"2024-03-26T05:22:09.990900+0000","last_fullsized":"2024-03-26T05:22:09.990900+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:47:45.134687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809857+0000","last_change":"2024-03-26T05:21:30.830415+0000","last_active":"2024-03-26T05:22:09.809857+0000","last_peered":"2024-03-26T05:22:09.809857+0000","last_clean":"2024-03-26T05:22:09.809857+0000","last_became_active":"2024-03-26T05:21:30.830091+0000","last_became_peered":"2024-03-26T05:21:30.830091+0000","last_unstale":"2024-03-26T05:22:09.809857+0000","last_undegraded":"2024-03-26T05:22:09.809857+0000","last_fullsized":"2024-03-26T05:22:09.809857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:06:33.998542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810699+0000","last_change":"2024-03-26T05:21:32.850623+0000","last_active":"2024-03-26T05:22:09.810699+0000","last_peered":"2024-03-26T05:22:09.810699+0000","last_clean":"2024-03-26T05:22:09.810699+0000","last_became_active":"2024-03-26T05:21:32.848371+0000","last_became_peered":"2024-03-26T05:21:32.848371+0000","last_unstale":"2024-03-26T05:22:09.810699+0000","last_undegraded":"2024-03-26T05:22:09.810699+0000","last_fullsized":"2024-03-26T05:22:09.810699+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:45:10.703984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992259+0000","last_change":"2024-03-26T05:21:36.859073+0000","last_active":"2024-03-26T05:22:09.992259+0000","last_peered":"2024-03-26T05:22:09.992259+0000","last_clean":"2024-03-26T05:22:09.992259+0000","last_became_active":"2024-03-26T05:21:36.858609+0000","last_became_peered":"2024-03-26T05:21:36.858609+0000","last_unstale":"2024-03-26T05:22:09.992259+0000","last_undegraded":"2024-03-26T05:22:09.992259+0000","last_fullsized":"2024-03-26T05:22:09.992259+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:18.608563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889826+0000","last_change":"2024-03-26T05:21:34.853358+0000","last_active":"2024-03-26T05:22:45.889826+0000","last_peered":"2024-03-26T05:22:45.889826+0000","last_clean":"2024-03-26T05:22:45.889826+0000","last_became_active":"2024-03-26T05:21:34.853068+0000","last_became_peered":"2024-03-26T05:21:34.853068+0000","last_unstale":"2024-03-26T05:22:45.889826+0000","last_undegraded":"2024-03-26T05:22:45.889826+0000","last_fullsized":"2024-03-26T05:22:45.889826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:00:31.335053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923166+0000","last_change":"2024-03-26T05:21:30.835120+0000","last_active":"2024-03-26T05:22:09.923166+0000","last_peered":"2024-03-26T05:22:09.923166+0000","last_clean":"2024-03-26T05:22:09.923166+0000","last_became_active":"2024-03-26T05:21:30.834834+0000","last_became_peered":"2024-03-26T05:21:30.834834+0000","last_unstale":"2024-03-26T05:22:09.923166+0000","last_undegraded":"2024-03-26T05:22:09.923166+0000","last_fullsized":"2024-03-26T05:22:09.923166+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:31:02.545407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'22","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807147+0000","last_change":"2024-03-26T05:21:33.316317+0000","last_active":"2024-03-26T05:22:09.807147+0000","last_peered":"2024-03-26T05:22:09.807147+0000","last_clean":"2024-03-26T05:22:09.807147+0000","last_became_active":"2024-03-26T05:21:33.316042+0000","last_became_peered":"2024-03-26T05:21:33.316042+0000","last_unstale":"2024-03-26T05:22:09.807147+0000","last_undegraded":"2024-03-26T05:22:09.807147+0000","last_fullsized":"2024-03-26T05:22:09.807147+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:10:12.931440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810349+0000","last_change":"2024-03-26T05:21:36.856309+0000","last_active":"2024-03-26T05:22:09.810349+0000","last_peered":"2024-03-26T05:22:09.810349+0000","last_clean":"2024-03-26T05:22:09.810349+0000","last_became_active":"2024-03-26T05:21:36.856030+0000","last_became_peered":"2024-03-26T05:21:36.856030+0000","last_unstale":"2024-03-26T05:22:09.810349+0000","last_undegraded":"2024-03-26T05:22:09.810349+0000","last_fullsized":"2024-03-26T05:22:09.810349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:20:36.064630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889507+0000","last_change":"2024-03-26T05:21:34.847671+0000","last_active":"2024-03-26T05:22:45.889507+0000","last_peered":"2024-03-26T05:22:45.889507+0000","last_clean":"2024-03-26T05:22:45.889507+0000","last_became_active":"2024-03-26T05:21:34.847169+0000","last_became_peered":"2024-03-26T05:21:34.847169+0000","last_unstale":"2024-03-26T05:22:45.889507+0000","last_undegraded":"2024-03-26T05:22:45.889507+0000","last_fullsized":"2024-03-26T05:22:45.889507+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:21:51.596952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991792+0000","last_change":"2024-03-26T05:21:30.827607+0000","last_active":"2024-03-26T05:22:09.991792+0000","last_peered":"2024-03-26T05:22:09.991792+0000","last_clean":"2024-03-26T05:22:09.991792+0000","last_became_active":"2024-03-26T05:21:30.827141+0000","last_became_peered":"2024-03-26T05:21:30.827141+0000","last_unstale":"2024-03-26T05:22:09.991792+0000","last_undegraded":"2024-03-26T05:22:09.991792+0000","last_fullsized":"2024-03-26T05:22:09.991792+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:25:39.858325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'15","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912629+0000","last_change":"2024-03-26T05:21:33.316834+0000","last_active":"2024-03-26T05:22:09.912629+0000","last_peered":"2024-03-26T05:22:09.912629+0000","last_clean":"2024-03-26T05:22:09.912629+0000","last_became_active":"2024-03-26T05:21:33.316512+0000","last_became_peered":"2024-03-26T05:21:33.316512+0000","last_unstale":"2024-03-26T05:22:09.912629+0000","last_undegraded":"2024-03-26T05:22:09.912629+0000","last_fullsized":"2024-03-26T05:22:09.912629+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:27:01.739178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807068+0000","last_change":"2024-03-26T05:21:36.859896+0000","last_active":"2024-03-26T05:22:09.807068+0000","last_peered":"2024-03-26T05:22:09.807068+0000","last_clean":"2024-03-26T05:22:09.807068+0000","last_became_active":"2024-03-26T05:21:36.859613+0000","last_became_peered":"2024-03-26T05:21:36.859613+0000","last_unstale":"2024-03-26T05:22:09.807068+0000","last_undegraded":"2024-03-26T05:22:09.807068+0000","last_fullsized":"2024-03-26T05:22:09.807068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:56:58.193646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808639+0000","last_change":"2024-03-26T05:21:34.849127+0000","last_active":"2024-03-26T05:22:09.808639+0000","last_peered":"2024-03-26T05:22:09.808639+0000","last_clean":"2024-03-26T05:22:09.808639+0000","last_became_active":"2024-03-26T05:21:34.848831+0000","last_became_peered":"2024-03-26T05:21:34.848831+0000","last_unstale":"2024-03-26T05:22:09.808639+0000","last_undegraded":"2024-03-26T05:22:09.808639+0000","last_fullsized":"2024-03-26T05:22:09.808639+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:41:33.435188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923106+0000","last_change":"2024-03-26T05:21:30.831882+0000","last_active":"2024-03-26T05:22:09.923106+0000","last_peered":"2024-03-26T05:22:09.923106+0000","last_clean":"2024-03-26T05:22:09.923106+0000","last_became_active":"2024-03-26T05:21:30.831563+0000","last_became_peered":"2024-03-26T05:21:30.831563+0000","last_unstale":"2024-03-26T05:22:09.923106+0000","last_undegraded":"2024-03-26T05:22:09.923106+0000","last_fullsized":"2024-03-26T05:22:09.923106+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:55:51.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'23","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992726+0000","last_change":"2024-03-26T05:21:33.315441+0000","last_active":"2024-03-26T05:22:09.992726+0000","last_peered":"2024-03-26T05:22:09.992726+0000","last_clean":"2024-03-26T05:22:09.992726+0000","last_became_active":"2024-03-26T05:21:33.315118+0000","last_became_peered":"2024-03-26T05:21:33.315118+0000","last_unstale":"2024-03-26T05:22:09.992726+0000","last_undegraded":"2024-03-26T05:22:09.992726+0000","last_fullsized":"2024-03-26T05:22:09.992726+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:08:35.526716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923462+0000","last_change":"2024-03-26T05:21:37.150099+0000","last_active":"2024-03-26T05:22:09.923462+0000","last_peered":"2024-03-26T05:22:09.923462+0000","last_clean":"2024-03-26T05:22:09.923462+0000","last_became_active":"2024-03-26T05:21:37.149768+0000","last_became_peered":"2024-03-26T05:21:37.149768+0000","last_unstale":"2024-03-26T05:22:09.923462+0000","last_undegraded":"2024-03-26T05:22:09.923462+0000","last_fullsized":"2024-03-26T05:22:09.923462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:35:45.876849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912517+0000","last_change":"2024-03-26T05:21:34.852183+0000","last_active":"2024-03-26T05:22:09.912517+0000","last_peered":"2024-03-26T05:22:09.912517+0000","last_clean":"2024-03-26T05:22:09.912517+0000","last_became_active":"2024-03-26T05:21:34.851872+0000","last_became_peered":"2024-03-26T05:21:34.851872+0000","last_unstale":"2024-03-26T05:22:09.912517+0000","last_undegraded":"2024-03-26T05:22:09.912517+0000","last_fullsized":"2024-03-26T05:22:09.912517+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:23:09.538518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990284+0000","last_change":"2024-03-26T05:21:30.826246+0000","last_active":"2024-03-26T05:22:09.990284+0000","last_peered":"2024-03-26T05:22:09.990284+0000","last_clean":"2024-03-26T05:22:09.990284+0000","last_became_active":"2024-03-26T05:21:30.825958+0000","last_became_peered":"2024-03-26T05:21:30.825958+0000","last_unstale":"2024-03-26T05:22:09.990284+0000","last_undegraded":"2024-03-26T05:22:09.990284+0000","last_fullsized":"2024-03-26T05:22:09.990284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:18.090816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'12","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808820+0000","last_change":"2024-03-26T05:21:33.316854+0000","last_active":"2024-03-26T05:22:09.808820+0000","last_peered":"2024-03-26T05:22:09.808820+0000","last_clean":"2024-03-26T05:22:09.808820+0000","last_became_active":"2024-03-26T05:21:33.316574+0000","last_became_peered":"2024-03-26T05:21:33.316574+0000","last_unstale":"2024-03-26T05:22:09.808820+0000","last_undegraded":"2024-03-26T05:22:09.808820+0000","last_fullsized":"2024-03-26T05:22:09.808820+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:49:25.559854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911922+0000","last_change":"2024-03-26T05:21:37.148965+0000","last_active":"2024-03-26T05:22:09.911922+0000","last_peered":"2024-03-26T05:22:09.911922+0000","last_clean":"2024-03-26T05:22:09.911922+0000","last_became_active":"2024-03-26T05:21:37.147966+0000","last_became_peered":"2024-03-26T05:21:37.147966+0000","last_unstale":"2024-03-26T05:22:09.911922+0000","last_undegraded":"2024-03-26T05:22:09.911922+0000","last_fullsized":"2024-03-26T05:22:09.911922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:47:52.078291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923930+0000","last_change":"2024-03-26T05:21:30.835220+0000","last_active":"2024-03-26T05:22:09.923930+0000","last_peered":"2024-03-26T05:22:09.923930+0000","last_clean":"2024-03-26T05:22:09.923930+0000","last_became_active":"2024-03-26T05:21:30.834868+0000","last_became_peered":"2024-03-26T05:21:30.834868+0000","last_unstale":"2024-03-26T05:22:09.923930+0000","last_undegraded":"2024-03-26T05:22:09.923930+0000","last_fullsized":"2024-03-26T05:22:09.923930+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:38:22.083221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'27","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911548+0000","last_change":"2024-03-26T05:21:33.316316+0000","last_active":"2024-03-26T05:22:09.911548+0000","last_peered":"2024-03-26T05:22:09.911548+0000","last_clean":"2024-03-26T05:22:09.911548+0000","last_became_active":"2024-03-26T05:21:33.315943+0000","last_became_peered":"2024-03-26T05:21:33.315943+0000","last_unstale":"2024-03-26T05:22:09.911548+0000","last_undegraded":"2024-03-26T05:22:09.911548+0000","last_fullsized":"2024-03-26T05:22:09.911548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:56:04.964323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889821+0000","last_change":"2024-03-26T05:21:34.849489+0000","last_active":"2024-03-26T05:22:45.889821+0000","last_peered":"2024-03-26T05:22:45.889821+0000","last_clean":"2024-03-26T05:22:45.889821+0000","last_became_active":"2024-03-26T05:21:34.849194+0000","last_became_peered":"2024-03-26T05:21:34.849194+0000","last_unstale":"2024-03-26T05:22:45.889821+0000","last_undegraded":"2024-03-26T05:22:45.889821+0000","last_fullsized":"2024-03-26T05:22:45.889821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:52.786092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807295+0000","last_change":"2024-03-26T05:21:36.853858+0000","last_active":"2024-03-26T05:22:09.807295+0000","last_peered":"2024-03-26T05:22:09.807295+0000","last_clean":"2024-03-26T05:22:09.807295+0000","last_became_active":"2024-03-26T05:21:36.853556+0000","last_became_peered":"2024-03-26T05:21:36.853556+0000","last_unstale":"2024-03-26T05:22:09.807295+0000","last_undegraded":"2024-03-26T05:22:09.807295+0000","last_fullsized":"2024-03-26T05:22:09.807295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:00:51.819093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923805+0000","last_change":"2024-03-26T05:21:30.834326+0000","last_active":"2024-03-26T05:22:09.923805+0000","last_peered":"2024-03-26T05:22:09.923805+0000","last_clean":"2024-03-26T05:22:09.923805+0000","last_became_active":"2024-03-26T05:21:30.833967+0000","last_became_peered":"2024-03-26T05:21:30.833967+0000","last_unstale":"2024-03-26T05:22:09.923805+0000","last_undegraded":"2024-03-26T05:22:09.923805+0000","last_fullsized":"2024-03-26T05:22:09.923805+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:18:03.676676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'24","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809274+0000","last_change":"2024-03-26T05:21:32.850489+0000","last_active":"2024-03-26T05:22:09.809274+0000","last_peered":"2024-03-26T05:22:09.809274+0000","last_clean":"2024-03-26T05:22:09.809274+0000","last_became_active":"2024-03-26T05:21:32.850120+0000","last_became_peered":"2024-03-26T05:21:32.850120+0000","last_unstale":"2024-03-26T05:22:09.809274+0000","last_undegraded":"2024-03-26T05:22:09.809274+0000","last_fullsized":"2024-03-26T05:22:09.809274+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:23:04.138636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.805298+0000","last_change":"2024-03-26T05:21:34.854951+0000","last_active":"2024-03-26T05:22:09.805298+0000","last_peered":"2024-03-26T05:22:09.805298+0000","last_clean":"2024-03-26T05:22:09.805298+0000","last_became_active":"2024-03-26T05:21:34.854626+0000","last_became_peered":"2024-03-26T05:21:34.854626+0000","last_unstale":"2024-03-26T05:22:09.805298+0000","last_undegraded":"2024-03-26T05:22:09.805298+0000","last_fullsized":"2024-03-26T05:22:09.805298+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:14:39.612969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.922880+0000","last_change":"2024-03-26T05:21:36.856466+0000","last_active":"2024-03-26T05:22:09.922880+0000","last_peered":"2024-03-26T05:22:09.922880+0000","last_clean":"2024-03-26T05:22:09.922880+0000","last_became_active":"2024-03-26T05:21:36.856184+0000","last_became_peered":"2024-03-26T05:21:36.856184+0000","last_unstale":"2024-03-26T05:22:09.922880+0000","last_undegraded":"2024-03-26T05:22:09.922880+0000","last_fullsized":"2024-03-26T05:22:09.922880+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:14:33.325922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806285+0000","last_change":"2024-03-26T05:21:34.853291+0000","last_active":"2024-03-26T05:22:09.806285+0000","last_peered":"2024-03-26T05:22:09.806285+0000","last_clean":"2024-03-26T05:22:09.806285+0000","last_became_active":"2024-03-26T05:21:34.852985+0000","last_became_peered":"2024-03-26T05:21:34.852985+0000","last_unstale":"2024-03-26T05:22:09.806285+0000","last_undegraded":"2024-03-26T05:22:09.806285+0000","last_fullsized":"2024-03-26T05:22:09.806285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:27:32.396824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923671+0000","last_change":"2024-03-26T05:21:30.829712+0000","last_active":"2024-03-26T05:22:09.923671+0000","last_peered":"2024-03-26T05:22:09.923671+0000","last_clean":"2024-03-26T05:22:09.923671+0000","last_became_active":"2024-03-26T05:21:30.829418+0000","last_became_peered":"2024-03-26T05:21:30.829418+0000","last_unstale":"2024-03-26T05:22:09.923671+0000","last_undegraded":"2024-03-26T05:22:09.923671+0000","last_fullsized":"2024-03-26T05:22:09.923671+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:02:34.857021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'25","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990802+0000","last_change":"2024-03-26T05:21:33.318417+0000","last_active":"2024-03-26T05:22:09.990802+0000","last_peered":"2024-03-26T05:22:09.990802+0000","last_clean":"2024-03-26T05:22:09.990802+0000","last_became_active":"2024-03-26T05:21:33.317980+0000","last_became_peered":"2024-03-26T05:21:33.317980+0000","last_unstale":"2024-03-26T05:22:09.990802+0000","last_undegraded":"2024-03-26T05:22:09.990802+0000","last_fullsized":"2024-03-26T05:22:09.990802+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:14:57.144778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991684+0000","last_change":"2024-03-26T05:21:36.858892+0000","last_active":"2024-03-26T05:22:09.991684+0000","last_peered":"2024-03-26T05:22:09.991684+0000","last_clean":"2024-03-26T05:22:09.991684+0000","last_became_active":"2024-03-26T05:21:36.858505+0000","last_became_peered":"2024-03-26T05:21:36.858505+0000","last_unstale":"2024-03-26T05:22:09.991684+0000","last_undegraded":"2024-03-26T05:22:09.991684+0000","last_fullsized":"2024-03-26T05:22:09.991684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:39:08.552790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.805938+0000","last_change":"2024-03-26T05:21:34.842418+0000","last_active":"2024-03-26T05:22:09.805938+0000","last_peered":"2024-03-26T05:22:09.805938+0000","last_clean":"2024-03-26T05:22:09.805938+0000","last_became_active":"2024-03-26T05:21:34.842126+0000","last_became_peered":"2024-03-26T05:21:34.842126+0000","last_unstale":"2024-03-26T05:22:09.805938+0000","last_undegraded":"2024-03-26T05:22:09.805938+0000","last_fullsized":"2024-03-26T05:22:09.805938+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:18:10.305337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911963+0000","last_change":"2024-03-26T05:21:30.826423+0000","last_active":"2024-03-26T05:22:09.911963+0000","last_peered":"2024-03-26T05:22:09.911963+0000","last_clean":"2024-03-26T05:22:09.911963+0000","last_became_active":"2024-03-26T05:21:30.826094+0000","last_became_peered":"2024-03-26T05:21:30.826094+0000","last_unstale":"2024-03-26T05:22:09.911963+0000","last_undegraded":"2024-03-26T05:22:09.911963+0000","last_fullsized":"2024-03-26T05:22:09.911963+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:37:27.454663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'15","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992404+0000","last_change":"2024-03-26T05:21:57.699762+0000","last_active":"2024-03-26T05:22:09.992404+0000","last_peered":"2024-03-26T05:22:09.992404+0000","last_clean":"2024-03-26T05:22:09.992404+0000","last_became_active":"2024-03-26T05:21:57.687329+0000","last_became_peered":"2024-03-26T05:21:57.687329+0000","last_unstale":"2024-03-26T05:22:09.992404+0000","last_undegraded":"2024-03-26T05:22:09.992404+0000","last_fullsized":"2024-03-26T05:22:09.992404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:32:31.194632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807076+0000","last_change":"2024-03-26T05:21:36.860966+0000","last_active":"2024-03-26T05:22:09.807076+0000","last_peered":"2024-03-26T05:22:09.807076+0000","last_clean":"2024-03-26T05:22:09.807076+0000","last_became_active":"2024-03-26T05:21:36.860629+0000","last_became_peered":"2024-03-26T05:21:36.860629+0000","last_unstale":"2024-03-26T05:22:09.807076+0000","last_undegraded":"2024-03-26T05:22:09.807076+0000","last_fullsized":"2024-03-26T05:22:09.807076+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:10:07.550878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911668+0000","last_change":"2024-03-26T05:21:33.839958+0000","last_active":"2024-03-26T05:22:09.911668+0000","last_peered":"2024-03-26T05:22:09.911668+0000","last_clean":"2024-03-26T05:22:09.911668+0000","last_became_active":"2024-03-26T05:21:29.817501+0000","last_became_peered":"2024-03-26T05:21:29.817501+0000","last_unstale":"2024-03-26T05:22:09.911668+0000","last_undegraded":"2024-03-26T05:22:09.911668+0000","last_fullsized":"2024-03-26T05:22:09.911668+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_clean_scrub_stamp":"2024-03-26T05:21:28.796453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:13:34.952002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052850800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809189+0000","last_change":"2024-03-26T05:21:30.826660+0000","last_active":"2024-03-26T05:22:09.809189+0000","last_peered":"2024-03-26T05:22:09.809189+0000","last_clean":"2024-03-26T05:22:09.809189+0000","last_became_active":"2024-03-26T05:21:30.826338+0000","last_became_peered":"2024-03-26T05:21:30.826338+0000","last_unstale":"2024-03-26T05:22:09.809189+0000","last_undegraded":"2024-03-26T05:22:09.809189+0000","last_fullsized":"2024-03-26T05:22:09.809189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:05:54.055875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'34","reported_seq":106,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889437+0000","last_change":"2024-03-26T05:21:32.847971+0000","last_active":"2024-03-26T05:22:45.889437+0000","last_peered":"2024-03-26T05:22:45.889437+0000","last_clean":"2024-03-26T05:22:45.889437+0000","last_became_active":"2024-03-26T05:21:32.847624+0000","last_became_peered":"2024-03-26T05:21:32.847624+0000","last_unstale":"2024-03-26T05:22:45.889437+0000","last_undegraded":"2024-03-26T05:22:45.889437+0000","last_fullsized":"2024-03-26T05:22:45.889437+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:17:46.448596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992159+0000","last_change":"2024-03-26T05:21:34.852873+0000","last_active":"2024-03-26T05:22:09.992159+0000","last_peered":"2024-03-26T05:22:09.992159+0000","last_clean":"2024-03-26T05:22:09.992159+0000","last_became_active":"2024-03-26T05:21:34.852506+0000","last_became_peered":"2024-03-26T05:21:34.852506+0000","last_unstale":"2024-03-26T05:22:09.992159+0000","last_undegraded":"2024-03-26T05:22:09.992159+0000","last_fullsized":"2024-03-26T05:22:09.992159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:15:28.753927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923756+0000","last_change":"2024-03-26T05:21:37.150798+0000","last_active":"2024-03-26T05:22:09.923756+0000","last_peered":"2024-03-26T05:22:09.923756+0000","last_clean":"2024-03-26T05:22:09.923756+0000","last_became_active":"2024-03-26T05:21:37.150491+0000","last_became_peered":"2024-03-26T05:21:37.150491+0000","last_unstale":"2024-03-26T05:22:09.923756+0000","last_undegraded":"2024-03-26T05:22:09.923756+0000","last_fullsized":"2024-03-26T05:22:09.923756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:00.686155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990625+0000","last_change":"2024-03-26T05:21:30.829356+0000","last_active":"2024-03-26T05:22:09.990625+0000","last_peered":"2024-03-26T05:22:09.990625+0000","last_clean":"2024-03-26T05:22:09.990625+0000","last_became_active":"2024-03-26T05:21:30.829035+0000","last_became_peered":"2024-03-26T05:21:30.829035+0000","last_unstale":"2024-03-26T05:22:09.990625+0000","last_undegraded":"2024-03-26T05:22:09.990625+0000","last_fullsized":"2024-03-26T05:22:09.990625+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:56:02.276546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.805585+0000","last_change":"2024-03-26T05:21:33.835345+0000","last_active":"2024-03-26T05:22:09.805585+0000","last_peered":"2024-03-26T05:22:09.805585+0000","last_clean":"2024-03-26T05:22:09.805585+0000","last_became_active":"2024-03-26T05:21:29.813603+0000","last_became_peered":"2024-03-26T05:21:29.813603+0000","last_unstale":"2024-03-26T05:22:09.805585+0000","last_undegraded":"2024-03-26T05:22:09.805585+0000","last_fullsized":"2024-03-26T05:22:09.805585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_clean_scrub_stamp":"2024-03-26T05:21:28.796453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:31:14.841877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053699499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'19","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991908+0000","last_change":"2024-03-26T05:21:32.851263+0000","last_active":"2024-03-26T05:22:09.991908+0000","last_peered":"2024-03-26T05:22:09.991908+0000","last_clean":"2024-03-26T05:22:09.991908+0000","last_became_active":"2024-03-26T05:21:32.849895+0000","last_became_peered":"2024-03-26T05:21:32.849895+0000","last_unstale":"2024-03-26T05:22:09.991908+0000","last_undegraded":"2024-03-26T05:22:09.991908+0000","last_fullsized":"2024-03-26T05:22:09.991908+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:09:57.668003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.805353+0000","last_change":"2024-03-26T05:21:34.850606+0000","last_active":"2024-03-26T05:22:09.805353+0000","last_peered":"2024-03-26T05:22:09.805353+0000","last_clean":"2024-03-26T05:22:09.805353+0000","last_became_active":"2024-03-26T05:21:34.850299+0000","last_became_peered":"2024-03-26T05:21:34.850299+0000","last_unstale":"2024-03-26T05:22:09.805353+0000","last_undegraded":"2024-03-26T05:22:09.805353+0000","last_fullsized":"2024-03-26T05:22:09.805353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:57.004815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912186+0000","last_change":"2024-03-26T05:21:36.857575+0000","last_active":"2024-03-26T05:22:09.912186+0000","last_peered":"2024-03-26T05:22:09.912186+0000","last_clean":"2024-03-26T05:22:09.912186+0000","last_became_active":"2024-03-26T05:21:36.857203+0000","last_became_peered":"2024-03-26T05:21:36.857203+0000","last_unstale":"2024-03-26T05:22:09.912186+0000","last_undegraded":"2024-03-26T05:22:09.912186+0000","last_fullsized":"2024-03-26T05:22:09.912186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:05:37.683858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":278,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.901478+0000","last_change":"2024-03-26T05:21:21.195182+0000","last_active":"2024-03-26T05:22:11.901478+0000","last_peered":"2024-03-26T05:22:11.901478+0000","last_clean":"2024-03-26T05:22:11.901478+0000","last_became_active":"2024-03-26T05:21:21.194654+0000","last_became_peered":"2024-03-26T05:21:21.194654+0000","last_unstale":"2024-03-26T05:22:11.901478+0000","last_undegraded":"2024-03-26T05:22:11.901478+0000","last_fullsized":"2024-03-26T05:22:11.901478+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:15.266592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:19:15.266592+0000","last_clean_scrub_stamp":"2024-03-26T05:19:15.266592+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:47.295096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808019+0000","last_change":"2024-03-26T05:21:30.825786+0000","last_active":"2024-03-26T05:22:09.808019+0000","last_peered":"2024-03-26T05:22:09.808019+0000","last_clean":"2024-03-26T05:22:09.808019+0000","last_became_active":"2024-03-26T05:21:30.825516+0000","last_became_peered":"2024-03-26T05:21:30.825516+0000","last_unstale":"2024-03-26T05:22:09.808019+0000","last_undegraded":"2024-03-26T05:22:09.808019+0000","last_fullsized":"2024-03-26T05:22:09.808019+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:00:14.707568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.890108+0000","last_change":"2024-03-26T05:21:33.315830+0000","last_active":"2024-03-26T05:22:45.890108+0000","last_peered":"2024-03-26T05:22:45.890108+0000","last_clean":"2024-03-26T05:22:45.890108+0000","last_became_active":"2024-03-26T05:21:33.315413+0000","last_became_peered":"2024-03-26T05:21:33.315413+0000","last_unstale":"2024-03-26T05:22:45.890108+0000","last_undegraded":"2024-03-26T05:22:45.890108+0000","last_fullsized":"2024-03-26T05:22:45.890108+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:08:58.855063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923486+0000","last_change":"2024-03-26T05:21:34.853592+0000","last_active":"2024-03-26T05:22:09.923486+0000","last_peered":"2024-03-26T05:22:09.923486+0000","last_clean":"2024-03-26T05:22:09.923486+0000","last_became_active":"2024-03-26T05:21:34.852461+0000","last_became_peered":"2024-03-26T05:21:34.852461+0000","last_unstale":"2024-03-26T05:22:09.923486+0000","last_undegraded":"2024-03-26T05:22:09.923486+0000","last_fullsized":"2024-03-26T05:22:09.923486+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:55:28.147166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991003+0000","last_change":"2024-03-26T05:21:36.857365+0000","last_active":"2024-03-26T05:22:09.991003+0000","last_peered":"2024-03-26T05:22:09.991003+0000","last_clean":"2024-03-26T05:22:09.991003+0000","last_became_active":"2024-03-26T05:21:36.856242+0000","last_became_peered":"2024-03-26T05:21:36.856242+0000","last_unstale":"2024-03-26T05:22:09.991003+0000","last_undegraded":"2024-03-26T05:22:09.991003+0000","last_fullsized":"2024-03-26T05:22:09.991003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:15:05.564582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991925+0000","last_change":"2024-03-26T05:21:30.832001+0000","last_active":"2024-03-26T05:22:09.991925+0000","last_peered":"2024-03-26T05:22:09.991925+0000","last_clean":"2024-03-26T05:22:09.991925+0000","last_became_active":"2024-03-26T05:21:30.831107+0000","last_became_peered":"2024-03-26T05:21:30.831107+0000","last_unstale":"2024-03-26T05:22:09.991925+0000","last_undegraded":"2024-03-26T05:22:09.991925+0000","last_fullsized":"2024-03-26T05:22:09.991925+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:26:55.425511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:53.935153+0000","last_change":"2024-03-26T05:21:33.844834+0000","last_active":"2024-03-26T05:22:53.935153+0000","last_peered":"2024-03-26T05:22:53.935153+0000","last_clean":"2024-03-26T05:22:53.935153+0000","last_became_active":"2024-03-26T05:21:29.818246+0000","last_became_peered":"2024-03-26T05:21:29.818246+0000","last_unstale":"2024-03-26T05:22:53.935153+0000","last_undegraded":"2024-03-26T05:22:53.935153+0000","last_fullsized":"2024-03-26T05:22:53.935153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_clean_scrub_stamp":"2024-03-26T05:21:28.796453+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:34:55.722020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000459813,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992356+0000","last_change":"2024-03-26T05:21:32.848362+0000","last_active":"2024-03-26T05:22:09.992356+0000","last_peered":"2024-03-26T05:22:09.992356+0000","last_clean":"2024-03-26T05:22:09.992356+0000","last_became_active":"2024-03-26T05:21:32.848080+0000","last_became_peered":"2024-03-26T05:21:32.848080+0000","last_unstale":"2024-03-26T05:22:09.992356+0000","last_undegraded":"2024-03-26T05:22:09.992356+0000","last_fullsized":"2024-03-26T05:22:09.992356+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:59:10.757138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912618+0000","last_change":"2024-03-26T05:21:34.855388+0000","last_active":"2024-03-26T05:22:09.912618+0000","last_peered":"2024-03-26T05:22:09.912618+0000","last_clean":"2024-03-26T05:22:09.912618+0000","last_became_active":"2024-03-26T05:21:34.855073+0000","last_became_peered":"2024-03-26T05:21:34.855073+0000","last_unstale":"2024-03-26T05:22:09.912618+0000","last_undegraded":"2024-03-26T05:22:09.912618+0000","last_fullsized":"2024-03-26T05:22:09.912618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:48:15.631848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990457+0000","last_change":"2024-03-26T05:21:37.148788+0000","last_active":"2024-03-26T05:22:09.990457+0000","last_peered":"2024-03-26T05:22:09.990457+0000","last_clean":"2024-03-26T05:22:09.990457+0000","last_became_active":"2024-03-26T05:21:37.148328+0000","last_became_peered":"2024-03-26T05:21:37.148328+0000","last_unstale":"2024-03-26T05:22:09.990457+0000","last_undegraded":"2024-03-26T05:22:09.990457+0000","last_fullsized":"2024-03-26T05:22:09.990457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:34:21.274218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991449+0000","last_change":"2024-03-26T05:21:30.824900+0000","last_active":"2024-03-26T05:22:09.991449+0000","last_peered":"2024-03-26T05:22:09.991449+0000","last_clean":"2024-03-26T05:22:09.991449+0000","last_became_active":"2024-03-26T05:21:30.824505+0000","last_became_peered":"2024-03-26T05:21:30.824505+0000","last_unstale":"2024-03-26T05:22:09.991449+0000","last_undegraded":"2024-03-26T05:22:09.991449+0000","last_fullsized":"2024-03-26T05:22:09.991449+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:10:01.721083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'26","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992737+0000","last_change":"2024-03-26T05:21:33.315461+0000","last_active":"2024-03-26T05:22:09.992737+0000","last_peered":"2024-03-26T05:22:09.992737+0000","last_clean":"2024-03-26T05:22:09.992737+0000","last_became_active":"2024-03-26T05:21:33.315097+0000","last_became_peered":"2024-03-26T05:21:33.315097+0000","last_unstale":"2024-03-26T05:22:09.992737+0000","last_undegraded":"2024-03-26T05:22:09.992737+0000","last_fullsized":"2024-03-26T05:22:09.992737+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:48:50.911641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990804+0000","last_change":"2024-03-26T05:21:34.852403+0000","last_active":"2024-03-26T05:22:09.990804+0000","last_peered":"2024-03-26T05:22:09.990804+0000","last_clean":"2024-03-26T05:22:09.990804+0000","last_became_active":"2024-03-26T05:21:34.852110+0000","last_became_peered":"2024-03-26T05:21:34.852110+0000","last_unstale":"2024-03-26T05:22:09.990804+0000","last_undegraded":"2024-03-26T05:22:09.990804+0000","last_fullsized":"2024-03-26T05:22:09.990804+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:42:10.729353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991074+0000","last_change":"2024-03-26T05:21:36.856082+0000","last_active":"2024-03-26T05:22:09.991074+0000","last_peered":"2024-03-26T05:22:09.991074+0000","last_clean":"2024-03-26T05:22:09.991074+0000","last_became_active":"2024-03-26T05:21:36.855787+0000","last_became_peered":"2024-03-26T05:21:36.855787+0000","last_unstale":"2024-03-26T05:22:09.991074+0000","last_undegraded":"2024-03-26T05:22:09.991074+0000","last_fullsized":"2024-03-26T05:22:09.991074+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:08:46.655147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912215+0000","last_change":"2024-03-26T05:21:30.819467+0000","last_active":"2024-03-26T05:22:09.912215+0000","last_peered":"2024-03-26T05:22:09.912215+0000","last_clean":"2024-03-26T05:22:09.912215+0000","last_became_active":"2024-03-26T05:21:30.819175+0000","last_became_peered":"2024-03-26T05:21:30.819175+0000","last_unstale":"2024-03-26T05:22:09.912215+0000","last_undegraded":"2024-03-26T05:22:09.912215+0000","last_fullsized":"2024-03-26T05:22:09.912215+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:04:11.909175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'16","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991021+0000","last_change":"2024-03-26T05:21:32.848936+0000","last_active":"2024-03-26T05:22:09.991021+0000","last_peered":"2024-03-26T05:22:09.991021+0000","last_clean":"2024-03-26T05:22:09.991021+0000","last_became_active":"2024-03-26T05:21:32.848369+0000","last_became_peered":"2024-03-26T05:21:32.848369+0000","last_unstale":"2024-03-26T05:22:09.991021+0000","last_undegraded":"2024-03-26T05:22:09.991021+0000","last_fullsized":"2024-03-26T05:22:09.991021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:04:00.986624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889450+0000","last_change":"2024-03-26T05:21:34.852447+0000","last_active":"2024-03-26T05:22:45.889450+0000","last_peered":"2024-03-26T05:22:45.889450+0000","last_clean":"2024-03-26T05:22:45.889450+0000","last_became_active":"2024-03-26T05:21:34.852034+0000","last_became_peered":"2024-03-26T05:21:34.852034+0000","last_unstale":"2024-03-26T05:22:45.889450+0000","last_undegraded":"2024-03-26T05:22:45.889450+0000","last_fullsized":"2024-03-26T05:22:45.889450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:36:08.950749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923397+0000","last_change":"2024-03-26T05:21:37.150760+0000","last_active":"2024-03-26T05:22:09.923397+0000","last_peered":"2024-03-26T05:22:09.923397+0000","last_clean":"2024-03-26T05:22:09.923397+0000","last_became_active":"2024-03-26T05:21:37.150451+0000","last_became_peered":"2024-03-26T05:21:37.150451+0000","last_unstale":"2024-03-26T05:22:09.923397+0000","last_undegraded":"2024-03-26T05:22:09.923397+0000","last_fullsized":"2024-03-26T05:22:09.923397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:50:55.067073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992492+0000","last_change":"2024-03-26T05:21:30.832264+0000","last_active":"2024-03-26T05:22:09.992492+0000","last_peered":"2024-03-26T05:22:09.992492+0000","last_clean":"2024-03-26T05:22:09.992492+0000","last_became_active":"2024-03-26T05:21:30.831939+0000","last_became_peered":"2024-03-26T05:21:30.831939+0000","last_unstale":"2024-03-26T05:22:09.992492+0000","last_undegraded":"2024-03-26T05:22:09.992492+0000","last_fullsized":"2024-03-26T05:22:09.992492+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:15:29.806546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'20","reported_seq":62,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809012+0000","last_change":"2024-03-26T05:21:32.843704+0000","last_active":"2024-03-26T05:22:09.809012+0000","last_peered":"2024-03-26T05:22:09.809012+0000","last_clean":"2024-03-26T05:22:09.809012+0000","last_became_active":"2024-03-26T05:21:32.843350+0000","last_became_peered":"2024-03-26T05:21:32.843350+0000","last_unstale":"2024-03-26T05:22:09.809012+0000","last_undegraded":"2024-03-26T05:22:09.809012+0000","last_fullsized":"2024-03-26T05:22:09.809012+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:07:38.742244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889435+0000","last_change":"2024-03-26T05:21:34.845930+0000","last_active":"2024-03-26T05:22:45.889435+0000","last_peered":"2024-03-26T05:22:45.889435+0000","last_clean":"2024-03-26T05:22:45.889435+0000","last_became_active":"2024-03-26T05:21:34.845606+0000","last_became_peered":"2024-03-26T05:21:34.845606+0000","last_unstale":"2024-03-26T05:22:45.889435+0000","last_undegraded":"2024-03-26T05:22:45.889435+0000","last_fullsized":"2024-03-26T05:22:45.889435+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:12:25.747607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808826+0000","last_change":"2024-03-26T05:21:36.856977+0000","last_active":"2024-03-26T05:22:09.808826+0000","last_peered":"2024-03-26T05:22:09.808826+0000","last_clean":"2024-03-26T05:22:09.808826+0000","last_became_active":"2024-03-26T05:21:36.856677+0000","last_became_peered":"2024-03-26T05:21:36.856677+0000","last_unstale":"2024-03-26T05:22:09.808826+0000","last_undegraded":"2024-03-26T05:22:09.808826+0000","last_fullsized":"2024-03-26T05:22:09.808826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:48:36.894140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809833+0000","last_change":"2024-03-26T05:21:30.826982+0000","last_active":"2024-03-26T05:22:09.809833+0000","last_peered":"2024-03-26T05:22:09.809833+0000","last_clean":"2024-03-26T05:22:09.809833+0000","last_became_active":"2024-03-26T05:21:30.826076+0000","last_became_peered":"2024-03-26T05:21:30.826076+0000","last_unstale":"2024-03-26T05:22:09.809833+0000","last_undegraded":"2024-03-26T05:22:09.809833+0000","last_fullsized":"2024-03-26T05:22:09.809833+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:23:19.558990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'30","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810173+0000","last_change":"2024-03-26T05:21:33.316202+0000","last_active":"2024-03-26T05:22:09.810173+0000","last_peered":"2024-03-26T05:22:09.810173+0000","last_clean":"2024-03-26T05:22:09.810173+0000","last_became_active":"2024-03-26T05:21:33.315916+0000","last_became_peered":"2024-03-26T05:21:33.315916+0000","last_unstale":"2024-03-26T05:22:09.810173+0000","last_undegraded":"2024-03-26T05:22:09.810173+0000","last_fullsized":"2024-03-26T05:22:09.810173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:45.078692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808736+0000","last_change":"2024-03-26T05:21:34.843885+0000","last_active":"2024-03-26T05:22:09.808736+0000","last_peered":"2024-03-26T05:22:09.808736+0000","last_clean":"2024-03-26T05:22:09.808736+0000","last_became_active":"2024-03-26T05:21:34.843557+0000","last_became_peered":"2024-03-26T05:21:34.843557+0000","last_unstale":"2024-03-26T05:22:09.808736+0000","last_undegraded":"2024-03-26T05:22:09.808736+0000","last_fullsized":"2024-03-26T05:22:09.808736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:57:50.589496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911826+0000","last_change":"2024-03-26T05:21:36.854664+0000","last_active":"2024-03-26T05:22:09.911826+0000","last_peered":"2024-03-26T05:22:09.911826+0000","last_clean":"2024-03-26T05:22:09.911826+0000","last_became_active":"2024-03-26T05:21:36.854402+0000","last_became_peered":"2024-03-26T05:21:36.854402+0000","last_unstale":"2024-03-26T05:22:09.911826+0000","last_undegraded":"2024-03-26T05:22:09.911826+0000","last_fullsized":"2024-03-26T05:22:09.911826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:26:51.369897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889821+0000","last_change":"2024-03-26T05:21:34.856402+0000","last_active":"2024-03-26T05:22:45.889821+0000","last_peered":"2024-03-26T05:22:45.889821+0000","last_clean":"2024-03-26T05:22:45.889821+0000","last_became_active":"2024-03-26T05:21:34.856106+0000","last_became_peered":"2024-03-26T05:21:34.856106+0000","last_unstale":"2024-03-26T05:22:45.889821+0000","last_undegraded":"2024-03-26T05:22:45.889821+0000","last_fullsized":"2024-03-26T05:22:45.889821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:17:05.578941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810087+0000","last_change":"2024-03-26T05:21:30.826413+0000","last_active":"2024-03-26T05:22:09.810087+0000","last_peered":"2024-03-26T05:22:09.810087+0000","last_clean":"2024-03-26T05:22:09.810087+0000","last_became_active":"2024-03-26T05:21:30.825733+0000","last_became_peered":"2024-03-26T05:21:30.825733+0000","last_unstale":"2024-03-26T05:22:09.810087+0000","last_undegraded":"2024-03-26T05:22:09.810087+0000","last_fullsized":"2024-03-26T05:22:09.810087+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:39:19.042121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991633+0000","last_change":"2024-03-26T05:21:32.853211+0000","last_active":"2024-03-26T05:22:09.991633+0000","last_peered":"2024-03-26T05:22:09.991633+0000","last_clean":"2024-03-26T05:22:09.991633+0000","last_became_active":"2024-03-26T05:21:32.852901+0000","last_became_peered":"2024-03-26T05:21:32.852901+0000","last_unstale":"2024-03-26T05:22:09.991633+0000","last_undegraded":"2024-03-26T05:22:09.991633+0000","last_fullsized":"2024-03-26T05:22:09.991633+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:06:01.791707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806763+0000","last_change":"2024-03-26T05:21:37.148681+0000","last_active":"2024-03-26T05:22:09.806763+0000","last_peered":"2024-03-26T05:22:09.806763+0000","last_clean":"2024-03-26T05:22:09.806763+0000","last_became_active":"2024-03-26T05:21:37.148395+0000","last_became_peered":"2024-03-26T05:21:37.148395+0000","last_unstale":"2024-03-26T05:22:09.806763+0000","last_undegraded":"2024-03-26T05:22:09.806763+0000","last_fullsized":"2024-03-26T05:22:09.806763+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:28:35.544419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912601+0000","last_change":"2024-03-26T05:21:34.846788+0000","last_active":"2024-03-26T05:22:09.912601+0000","last_peered":"2024-03-26T05:22:09.912601+0000","last_clean":"2024-03-26T05:22:09.912601+0000","last_became_active":"2024-03-26T05:21:34.846427+0000","last_became_peered":"2024-03-26T05:21:34.846427+0000","last_unstale":"2024-03-26T05:22:09.912601+0000","last_undegraded":"2024-03-26T05:22:09.912601+0000","last_fullsized":"2024-03-26T05:22:09.912601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:17:36.698608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808850+0000","last_change":"2024-03-26T05:21:30.831683+0000","last_active":"2024-03-26T05:22:09.808850+0000","last_peered":"2024-03-26T05:22:09.808850+0000","last_clean":"2024-03-26T05:22:09.808850+0000","last_became_active":"2024-03-26T05:21:30.830180+0000","last_became_peered":"2024-03-26T05:21:30.830180+0000","last_unstale":"2024-03-26T05:22:09.808850+0000","last_undegraded":"2024-03-26T05:22:09.808850+0000","last_fullsized":"2024-03-26T05:22:09.808850+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:02:15.235458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809603+0000","last_change":"2024-03-26T05:21:32.849241+0000","last_active":"2024-03-26T05:22:09.809603+0000","last_peered":"2024-03-26T05:22:09.809603+0000","last_clean":"2024-03-26T05:22:09.809603+0000","last_became_active":"2024-03-26T05:21:32.848484+0000","last_became_peered":"2024-03-26T05:21:32.848484+0000","last_unstale":"2024-03-26T05:22:09.809603+0000","last_undegraded":"2024-03-26T05:22:09.809603+0000","last_fullsized":"2024-03-26T05:22:09.809603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:49:47.299249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806644+0000","last_change":"2024-03-26T05:21:36.856534+0000","last_active":"2024-03-26T05:22:09.806644+0000","last_peered":"2024-03-26T05:22:09.806644+0000","last_clean":"2024-03-26T05:22:09.806644+0000","last_became_active":"2024-03-26T05:21:36.856230+0000","last_became_peered":"2024-03-26T05:21:36.856230+0000","last_unstale":"2024-03-26T05:22:09.806644+0000","last_undegraded":"2024-03-26T05:22:09.806644+0000","last_fullsized":"2024-03-26T05:22:09.806644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:52.728213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991784+0000","last_change":"2024-03-26T05:21:34.852423+0000","last_active":"2024-03-26T05:22:09.991784+0000","last_peered":"2024-03-26T05:22:09.991784+0000","last_clean":"2024-03-26T05:22:09.991784+0000","last_became_active":"2024-03-26T05:21:34.852117+0000","last_became_peered":"2024-03-26T05:21:34.852117+0000","last_unstale":"2024-03-26T05:22:09.991784+0000","last_undegraded":"2024-03-26T05:22:09.991784+0000","last_fullsized":"2024-03-26T05:22:09.991784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:34.664073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990677+0000","last_change":"2024-03-26T05:21:30.831931+0000","last_active":"2024-03-26T05:22:09.990677+0000","last_peered":"2024-03-26T05:22:09.990677+0000","last_clean":"2024-03-26T05:22:09.990677+0000","last_became_active":"2024-03-26T05:21:30.831627+0000","last_became_peered":"2024-03-26T05:21:30.831627+0000","last_unstale":"2024-03-26T05:22:09.990677+0000","last_undegraded":"2024-03-26T05:22:09.990677+0000","last_fullsized":"2024-03-26T05:22:09.990677+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:09:16.212042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810397+0000","last_change":"2024-03-26T05:21:32.849542+0000","last_active":"2024-03-26T05:22:09.810397+0000","last_peered":"2024-03-26T05:22:09.810397+0000","last_clean":"2024-03-26T05:22:09.810397+0000","last_became_active":"2024-03-26T05:21:32.848581+0000","last_became_peered":"2024-03-26T05:21:32.848581+0000","last_unstale":"2024-03-26T05:22:09.810397+0000","last_undegraded":"2024-03-26T05:22:09.810397+0000","last_fullsized":"2024-03-26T05:22:09.810397+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:39:35.696152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809947+0000","last_change":"2024-03-26T05:21:36.856981+0000","last_active":"2024-03-26T05:22:09.809947+0000","last_peered":"2024-03-26T05:22:09.809947+0000","last_clean":"2024-03-26T05:22:09.809947+0000","last_became_active":"2024-03-26T05:21:36.856638+0000","last_became_peered":"2024-03-26T05:21:36.856638+0000","last_unstale":"2024-03-26T05:22:09.809947+0000","last_undegraded":"2024-03-26T05:22:09.809947+0000","last_fullsized":"2024-03-26T05:22:09.809947+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:32:25.901388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889437+0000","last_change":"2024-03-26T05:21:34.848338+0000","last_active":"2024-03-26T05:22:45.889437+0000","last_peered":"2024-03-26T05:22:45.889437+0000","last_clean":"2024-03-26T05:22:45.889437+0000","last_became_active":"2024-03-26T05:21:34.848028+0000","last_became_peered":"2024-03-26T05:21:34.848028+0000","last_unstale":"2024-03-26T05:22:45.889437+0000","last_undegraded":"2024-03-26T05:22:45.889437+0000","last_fullsized":"2024-03-26T05:22:45.889437+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:46:31.513830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810463+0000","last_change":"2024-03-26T05:21:30.831172+0000","last_active":"2024-03-26T05:22:09.810463+0000","last_peered":"2024-03-26T05:22:09.810463+0000","last_clean":"2024-03-26T05:22:09.810463+0000","last_became_active":"2024-03-26T05:21:30.830830+0000","last_became_peered":"2024-03-26T05:21:30.830830+0000","last_unstale":"2024-03-26T05:22:09.810463+0000","last_undegraded":"2024-03-26T05:22:09.810463+0000","last_fullsized":"2024-03-26T05:22:09.810463+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:28:16.455447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809675+0000","last_change":"2024-03-26T05:21:32.843994+0000","last_active":"2024-03-26T05:22:09.809675+0000","last_peered":"2024-03-26T05:22:09.809675+0000","last_clean":"2024-03-26T05:22:09.809675+0000","last_became_active":"2024-03-26T05:21:32.843629+0000","last_became_peered":"2024-03-26T05:21:32.843629+0000","last_unstale":"2024-03-26T05:22:09.809675+0000","last_undegraded":"2024-03-26T05:22:09.809675+0000","last_fullsized":"2024-03-26T05:22:09.809675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:31:56.321038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":589,"ondisk_log_size":589,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32456,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712696,"statfs":{"total":95995035648,"available":95961800704,"internally_reserved":0,"allocated":1310720,"data_stored":872343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1298432,"data_stored":872179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":753664,"data_stored":282776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1314816,"data_stored":872085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":700416,"data_stored":281258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713244,"statfs":{"total":95995035648,"available":95962361856,"internally_reserved":0,"allocated":745472,"data_stored":283100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27732,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717420,"statfs":{"total":95995035648,"available":95966638080,"internally_reserved":0,"allocated":729088,"data_stored":281926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31876,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713276,"statfs":{"total":95995035648,"available":95962394624,"internally_reserved":0,"allocated":712704,"data_stored":282247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:22:58.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T05:22:58.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:57 smithi164 ceph-mon[33554]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:22:58.331 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:22:59.043 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:22:58 smithi164 ceph-mon[33554]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:59.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:22:58 smithi136 ceph-mon[38106]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:22:59.183 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:22:58 smithi136 ceph-mon[29560]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:00.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 -- 172.21.15.136:0/1716715020 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57581070f0 msgr2=0x7f57581074d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 --2- 172.21.15.136:0/1716715020 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57581070f0 0x7f57581074d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5750009f90 tx=0x7f575000b330 comp rx=0 tx=0).stop 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 -- 172.21.15.136:0/1716715020 shutdown_connections 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 --2- 172.21.15.136:0/1716715020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5758101030 0x7f575810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 --2- 172.21.15.136:0/1716715020 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57581070f0 0x7f57581074d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 --2- 172.21.15.136:0/1716715020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5758100670 0x7f5758100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 -- 172.21.15.136:0/1716715020 >> 172.21.15.136:0/1716715020 conn(0x7f5758077580 msgr2=0x7f57580fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 -- 172.21.15.136:0/1716715020 shutdown_connections 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.156+0000 7f575f658700 1 -- 172.21.15.136:0/1716715020 wait complete. 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575f658700 1 Processor -- start 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575f658700 1 -- start start 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575f658700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5758100670 0x7f5758197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:23:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575f658700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5758101030 0x7f5758198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575f658700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57581070f0 0x7f575819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575f658700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575819ced0 con 0x7f5758100670 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575f658700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575819d040 con 0x7f57581070f0 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575f658700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575819d200 con 0x7f5758101030 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575d3f4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5758100670 0x7f5758197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575cbf3700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5758101030 0x7f5758198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575cbf3700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5758101030 0x7f5758198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:57754/0 (socket says 172.21.15.136:57754) 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575d3f4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5758100670 0x7f5758197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:57112/0 (socket says 172.21.15.136:57112) 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.157+0000 7f575cbf3700 1 -- 172.21.15.136:0/484640684 learned_addr learned my addr 172.21.15.136:0/484640684 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575dbf5700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57581070f0 0x7f575819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575cbf3700 1 -- 172.21.15.136:0/484640684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57581070f0 msgr2=0x7f575819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575cbf3700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57581070f0 0x7f575819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575cbf3700 1 -- 172.21.15.136:0/484640684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5758100670 msgr2=0x7f5758197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:00.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575cbf3700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5758100670 0x7f5758197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575cbf3700 1 -- 172.21.15.136:0/484640684 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480095f0 con 0x7f5758101030 2024-03-26T05:23:00.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575d3f4700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5758100670 0x7f5758197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:23:00.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575cbf3700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5758101030 0x7f5758198530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f574800ec10 tx=0x7f574800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:23:00.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f574e7fc700 1 -- 172.21.15.136:0/484640684 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574800cd40 con 0x7f5758101030 2024-03-26T05:23:00.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5750009c40 con 0x7f5758101030 2024-03-26T05:23:00.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f574e7fc700 1 -- 172.21.15.136:0/484640684 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f57480044d0 con 0x7f5758101030 2024-03-26T05:23:00.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f574e7fc700 1 -- 172.21.15.136:0/484640684 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748010430 con 0x7f5758101030 2024-03-26T05:23:00.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.158+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57581a4b70 con 0x7f5758101030 2024-03-26T05:23:00.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.159+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5758103990 con 0x7f5758101030 2024-03-26T05:23:00.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.160+0000 7f574e7fc700 1 -- 172.21.15.136:0/484640684 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f5748003fe0 con 0x7f5758101030 2024-03-26T05:23:00.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.160+0000 7f574e7fc700 1 --2- 172.21.15.136:0/484640684 >> v2:172.21.15.136:6800/3380174081 conn(0x7f574406da80 0x7f574406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:23:00.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.160+0000 7f574e7fc700 1 -- 172.21.15.136:0/484640684 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f5748014070 con 0x7f5758101030 2024-03-26T05:23:00.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.160+0000 7f575d3f4700 1 --2- 172.21.15.136:0/484640684 >> v2:172.21.15.136:6800/3380174081 conn(0x7f574406da80 0x7f574406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:23:00.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.161+0000 7f575d3f4700 1 --2- 172.21.15.136:0/484640684 >> v2:172.21.15.136:6800/3380174081 conn(0x7f574406da80 0x7f574406ff40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5750009790 tx=0x7f57500096c0 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:23:00.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.165+0000 7f574e7fc700 1 -- 172.21.15.136:0/484640684 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f574805ad80 con 0x7f5758101030 2024-03-26T05:23:00.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.354+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 --> v2:172.21.15.136:6800/3380174081 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5758199350 con 0x7f574406da80 2024-03-26T05:23:00.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.364+0000 7f574e7fc700 1 -- 172.21.15.136:0/484640684 <== mgr.24485 v2:172.21.15.136:6800/3380174081 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346510 (secure 0 0 0) 0x7f5758199350 con 0x7f574406da80 2024-03-26T05:23:00.358 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:23:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 >> v2:172.21.15.136:6800/3380174081 conn(0x7f574406da80 msgr2=0x7f574406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 --2- 172.21.15.136:0/484640684 >> v2:172.21.15.136:6800/3380174081 conn(0x7f574406da80 0x7f574406ff40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5750009790 tx=0x7f57500096c0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5758101030 msgr2=0x7f5758198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5758101030 0x7f5758198530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f574800ec10 tx=0x7f574800ef20 comp rx=0 tx=0).stop 2024-03-26T05:23:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 shutdown_connections 2024-03-26T05:23:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 --2- 172.21.15.136:0/484640684 >> v2:172.21.15.136:6800/3380174081 conn(0x7f574406da80 0x7f574406ff40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57581070f0 0x7f575819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5758100670 0x7f5758197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 --2- 172.21.15.136:0/484640684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5758101030 0x7f5758198530 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:00.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 >> 172.21.15.136:0/484640684 conn(0x7f5758077580 msgr2=0x7f5758104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:23:00.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 shutdown_connections 2024-03-26T05:23:00.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:00.367+0000 7f575f658700 1 -- 172.21.15.136:0/484640684 wait complete. 2024-03-26T05:23:00.362 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-03-26T05:23:01.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:00 smithi136 ceph-mon[38106]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:01.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:00 smithi136 ceph-mon[38106]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:23:01.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:00 smithi136 ceph-mon[29560]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:01.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:00 smithi136 ceph-mon[29560]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:23:01.188 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-26T05:22:59.829041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1301,"num_read_kb":1115,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":796,"ondisk_log_size":796,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240180,"kb_used_data":7388,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721036,"statfs":{"total":767960285184,"available":767714340864,"internally_reserved":0,"allocated":7565312,"data_stored":4027914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":238276147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002952"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808748+0000","last_change":"2024-03-26T05:21:36.855842+0000","last_active":"2024-03-26T05:22:09.808748+0000","last_peered":"2024-03-26T05:22:09.808748+0000","last_clean":"2024-03-26T05:22:09.808748+0000","last_became_active":"2024-03-26T05:21:36.855239+0000","last_became_peered":"2024-03-26T05:21:36.855239+0000","last_unstale":"2024-03-26T05:22:09.808748+0000","last_undegraded":"2024-03-26T05:22:09.808748+0000","last_fullsized":"2024-03-26T05:22:09.808748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:42:47.079545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990571+0000","last_change":"2024-03-26T05:21:34.849980+0000","last_active":"2024-03-26T05:22:09.990571+0000","last_peered":"2024-03-26T05:22:09.990571+0000","last_clean":"2024-03-26T05:22:09.990571+0000","last_became_active":"2024-03-26T05:21:34.849655+0000","last_became_peered":"2024-03-26T05:21:34.849655+0000","last_unstale":"2024-03-26T05:22:09.990571+0000","last_undegraded":"2024-03-26T05:22:09.990571+0000","last_fullsized":"2024-03-26T05:22:09.990571+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:07:08.932699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992050+0000","last_change":"2024-03-26T05:21:30.831397+0000","last_active":"2024-03-26T05:22:09.992050+0000","last_peered":"2024-03-26T05:22:09.992050+0000","last_clean":"2024-03-26T05:22:09.992050+0000","last_became_active":"2024-03-26T05:21:30.831015+0000","last_became_peered":"2024-03-26T05:21:30.831015+0000","last_unstale":"2024-03-26T05:22:09.992050+0000","last_undegraded":"2024-03-26T05:22:09.992050+0000","last_fullsized":"2024-03-26T05:22:09.992050+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:58:57.861979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'16","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808921+0000","last_change":"2024-03-26T05:21:57.694124+0000","last_active":"2024-03-26T05:22:09.808921+0000","last_peered":"2024-03-26T05:22:09.808921+0000","last_clean":"2024-03-26T05:22:09.808921+0000","last_became_active":"2024-03-26T05:21:57.687312+0000","last_became_peered":"2024-03-26T05:21:57.687312+0000","last_unstale":"2024-03-26T05:22:09.808921+0000","last_undegraded":"2024-03-26T05:22:09.808921+0000","last_fullsized":"2024-03-26T05:22:09.808921+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:59:31.535805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807401+0000","last_change":"2024-03-26T05:21:37.149332+0000","last_active":"2024-03-26T05:22:09.807401+0000","last_peered":"2024-03-26T05:22:09.807401+0000","last_clean":"2024-03-26T05:22:09.807401+0000","last_became_active":"2024-03-26T05:21:37.149032+0000","last_became_peered":"2024-03-26T05:21:37.149032+0000","last_unstale":"2024-03-26T05:22:09.807401+0000","last_undegraded":"2024-03-26T05:22:09.807401+0000","last_fullsized":"2024-03-26T05:22:09.807401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:12:11.259814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808750+0000","last_change":"2024-03-26T05:21:34.847322+0000","last_active":"2024-03-26T05:22:09.808750+0000","last_peered":"2024-03-26T05:22:09.808750+0000","last_clean":"2024-03-26T05:22:09.808750+0000","last_became_active":"2024-03-26T05:21:34.847024+0000","last_became_peered":"2024-03-26T05:21:34.847024+0000","last_unstale":"2024-03-26T05:22:09.808750+0000","last_undegraded":"2024-03-26T05:22:09.808750+0000","last_fullsized":"2024-03-26T05:22:09.808750+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:00:10.781213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809505+0000","last_change":"2024-03-26T05:21:30.824036+0000","last_active":"2024-03-26T05:22:09.809505+0000","last_peered":"2024-03-26T05:22:09.809505+0000","last_clean":"2024-03-26T05:22:09.809505+0000","last_became_active":"2024-03-26T05:21:30.823731+0000","last_became_peered":"2024-03-26T05:21:30.823731+0000","last_unstale":"2024-03-26T05:22:09.809505+0000","last_undegraded":"2024-03-26T05:22:09.809505+0000","last_fullsized":"2024-03-26T05:22:09.809505+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:43:01.615281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'39","reported_seq":102,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806953+0000","last_change":"2024-03-26T05:21:32.847218+0000","last_active":"2024-03-26T05:22:09.806953+0000","last_peered":"2024-03-26T05:22:09.806953+0000","last_clean":"2024-03-26T05:22:09.806953+0000","last_became_active":"2024-03-26T05:21:32.846590+0000","last_became_peered":"2024-03-26T05:21:32.846590+0000","last_unstale":"2024-03-26T05:22:09.806953+0000","last_undegraded":"2024-03-26T05:22:09.806953+0000","last_fullsized":"2024-03-26T05:22:09.806953+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:49:18.147095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991232+0000","last_change":"2024-03-26T05:21:36.854407+0000","last_active":"2024-03-26T05:22:09.991232+0000","last_peered":"2024-03-26T05:22:09.991232+0000","last_clean":"2024-03-26T05:22:09.991232+0000","last_became_active":"2024-03-26T05:21:36.854053+0000","last_became_peered":"2024-03-26T05:21:36.854053+0000","last_unstale":"2024-03-26T05:22:09.991232+0000","last_undegraded":"2024-03-26T05:22:09.991232+0000","last_fullsized":"2024-03-26T05:22:09.991232+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:06:31.584429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911883+0000","last_change":"2024-03-26T05:21:34.853554+0000","last_active":"2024-03-26T05:22:09.911883+0000","last_peered":"2024-03-26T05:22:09.911883+0000","last_clean":"2024-03-26T05:22:09.911883+0000","last_became_active":"2024-03-26T05:21:34.853242+0000","last_became_peered":"2024-03-26T05:21:34.853242+0000","last_unstale":"2024-03-26T05:22:09.911883+0000","last_undegraded":"2024-03-26T05:22:09.911883+0000","last_fullsized":"2024-03-26T05:22:09.911883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:15:16.686440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911970+0000","last_change":"2024-03-26T05:21:30.831889+0000","last_active":"2024-03-26T05:22:09.911970+0000","last_peered":"2024-03-26T05:22:09.911970+0000","last_clean":"2024-03-26T05:22:09.911970+0000","last_became_active":"2024-03-26T05:21:30.830857+0000","last_became_peered":"2024-03-26T05:21:30.830857+0000","last_unstale":"2024-03-26T05:22:09.911970+0000","last_undegraded":"2024-03-26T05:22:09.911970+0000","last_fullsized":"2024-03-26T05:22:09.911970+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:41:15.266182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'21","reported_seq":69,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810847+0000","last_change":"2024-03-26T05:21:32.842463+0000","last_active":"2024-03-26T05:22:09.810847+0000","last_peered":"2024-03-26T05:22:09.810847+0000","last_clean":"2024-03-26T05:22:09.810847+0000","last_became_active":"2024-03-26T05:21:32.842119+0000","last_became_peered":"2024-03-26T05:21:32.842119+0000","last_unstale":"2024-03-26T05:22:09.810847+0000","last_undegraded":"2024-03-26T05:22:09.810847+0000","last_fullsized":"2024-03-26T05:22:09.810847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:24.029117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911566+0000","last_change":"2024-03-26T05:21:36.855058+0000","last_active":"2024-03-26T05:22:09.911566+0000","last_peered":"2024-03-26T05:22:09.911566+0000","last_clean":"2024-03-26T05:22:09.911566+0000","last_became_active":"2024-03-26T05:21:36.854745+0000","last_became_peered":"2024-03-26T05:21:36.854745+0000","last_unstale":"2024-03-26T05:22:09.911566+0000","last_undegraded":"2024-03-26T05:22:09.911566+0000","last_fullsized":"2024-03-26T05:22:09.911566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:53:04.473004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923015+0000","last_change":"2024-03-26T05:21:34.845990+0000","last_active":"2024-03-26T05:22:09.923015+0000","last_peered":"2024-03-26T05:22:09.923015+0000","last_clean":"2024-03-26T05:22:09.923015+0000","last_became_active":"2024-03-26T05:21:34.845529+0000","last_became_peered":"2024-03-26T05:21:34.845529+0000","last_unstale":"2024-03-26T05:22:09.923015+0000","last_undegraded":"2024-03-26T05:22:09.923015+0000","last_fullsized":"2024-03-26T05:22:09.923015+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:59:10.295747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911916+0000","last_change":"2024-03-26T05:21:30.831347+0000","last_active":"2024-03-26T05:22:09.911916+0000","last_peered":"2024-03-26T05:22:09.911916+0000","last_clean":"2024-03-26T05:22:09.911916+0000","last_became_active":"2024-03-26T05:21:30.831039+0000","last_became_peered":"2024-03-26T05:21:30.831039+0000","last_unstale":"2024-03-26T05:22:09.911916+0000","last_undegraded":"2024-03-26T05:22:09.911916+0000","last_fullsized":"2024-03-26T05:22:09.911916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:40:05.305903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'11","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809301+0000","last_change":"2024-03-26T05:21:32.849756+0000","last_active":"2024-03-26T05:22:09.809301+0000","last_peered":"2024-03-26T05:22:09.809301+0000","last_clean":"2024-03-26T05:22:09.809301+0000","last_became_active":"2024-03-26T05:21:32.848807+0000","last_became_peered":"2024-03-26T05:21:32.848807+0000","last_unstale":"2024-03-26T05:22:09.809301+0000","last_undegraded":"2024-03-26T05:22:09.809301+0000","last_fullsized":"2024-03-26T05:22:09.809301+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:52:55.525821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809733+0000","last_change":"2024-03-26T05:21:37.148267+0000","last_active":"2024-03-26T05:22:09.809733+0000","last_peered":"2024-03-26T05:22:09.809733+0000","last_clean":"2024-03-26T05:22:09.809733+0000","last_became_active":"2024-03-26T05:21:37.147971+0000","last_became_peered":"2024-03-26T05:21:37.147971+0000","last_unstale":"2024-03-26T05:22:09.809733+0000","last_undegraded":"2024-03-26T05:22:09.809733+0000","last_fullsized":"2024-03-26T05:22:09.809733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:03:32.847453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991521+0000","last_change":"2024-03-26T05:21:34.854477+0000","last_active":"2024-03-26T05:22:09.991521+0000","last_peered":"2024-03-26T05:22:09.991521+0000","last_clean":"2024-03-26T05:22:09.991521+0000","last_became_active":"2024-03-26T05:21:34.854052+0000","last_became_peered":"2024-03-26T05:21:34.854052+0000","last_unstale":"2024-03-26T05:22:09.991521+0000","last_undegraded":"2024-03-26T05:22:09.991521+0000","last_fullsized":"2024-03-26T05:22:09.991521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:56:19.159091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992646+0000","last_change":"2024-03-26T05:21:30.833534+0000","last_active":"2024-03-26T05:22:09.992646+0000","last_peered":"2024-03-26T05:22:09.992646+0000","last_clean":"2024-03-26T05:22:09.992646+0000","last_became_active":"2024-03-26T05:21:30.832166+0000","last_became_peered":"2024-03-26T05:21:30.832166+0000","last_unstale":"2024-03-26T05:22:09.992646+0000","last_undegraded":"2024-03-26T05:22:09.992646+0000","last_fullsized":"2024-03-26T05:22:09.992646+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:33:34.525001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807220+0000","last_change":"2024-03-26T05:21:57.805553+0000","last_active":"2024-03-26T05:22:09.807220+0000","last_peered":"2024-03-26T05:22:09.807220+0000","last_clean":"2024-03-26T05:22:09.807220+0000","last_became_active":"2024-03-26T05:21:57.796393+0000","last_became_peered":"2024-03-26T05:21:57.796393+0000","last_unstale":"2024-03-26T05:22:09.807220+0000","last_undegraded":"2024-03-26T05:22:09.807220+0000","last_fullsized":"2024-03-26T05:22:09.807220+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:43:24.576405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808727+0000","last_change":"2024-03-26T05:21:37.148629+0000","last_active":"2024-03-26T05:22:09.808727+0000","last_peered":"2024-03-26T05:22:09.808727+0000","last_clean":"2024-03-26T05:22:09.808727+0000","last_became_active":"2024-03-26T05:21:37.148268+0000","last_became_peered":"2024-03-26T05:21:37.148268+0000","last_unstale":"2024-03-26T05:22:09.808727+0000","last_undegraded":"2024-03-26T05:22:09.808727+0000","last_fullsized":"2024-03-26T05:22:09.808727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:25:35.698464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809582+0000","last_change":"2024-03-26T05:21:34.842701+0000","last_active":"2024-03-26T05:22:09.809582+0000","last_peered":"2024-03-26T05:22:09.809582+0000","last_clean":"2024-03-26T05:22:09.809582+0000","last_became_active":"2024-03-26T05:21:34.842427+0000","last_became_peered":"2024-03-26T05:21:34.842427+0000","last_unstale":"2024-03-26T05:22:09.809582+0000","last_undegraded":"2024-03-26T05:22:09.809582+0000","last_fullsized":"2024-03-26T05:22:09.809582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:53:25.426656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810982+0000","last_change":"2024-03-26T05:21:30.832724+0000","last_active":"2024-03-26T05:22:09.810982+0000","last_peered":"2024-03-26T05:22:09.810982+0000","last_clean":"2024-03-26T05:22:09.810982+0000","last_became_active":"2024-03-26T05:21:30.832143+0000","last_became_peered":"2024-03-26T05:21:30.832143+0000","last_unstale":"2024-03-26T05:22:09.810982+0000","last_undegraded":"2024-03-26T05:22:09.810982+0000","last_fullsized":"2024-03-26T05:22:09.810982+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:35:24.369382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'18","reported_seq":64,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807603+0000","last_change":"2024-03-26T05:21:32.850276+0000","last_active":"2024-03-26T05:22:09.807603+0000","last_peered":"2024-03-26T05:22:09.807603+0000","last_clean":"2024-03-26T05:22:09.807603+0000","last_became_active":"2024-03-26T05:21:32.849944+0000","last_became_peered":"2024-03-26T05:21:32.849944+0000","last_unstale":"2024-03-26T05:22:09.807603+0000","last_undegraded":"2024-03-26T05:22:09.807603+0000","last_fullsized":"2024-03-26T05:22:09.807603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:03:43.659584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923372+0000","last_change":"2024-03-26T05:21:36.857667+0000","last_active":"2024-03-26T05:22:09.923372+0000","last_peered":"2024-03-26T05:22:09.923372+0000","last_clean":"2024-03-26T05:22:09.923372+0000","last_became_active":"2024-03-26T05:21:36.857364+0000","last_became_peered":"2024-03-26T05:21:36.857364+0000","last_unstale":"2024-03-26T05:22:09.923372+0000","last_undegraded":"2024-03-26T05:22:09.923372+0000","last_fullsized":"2024-03-26T05:22:09.923372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:39:00.764868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889827+0000","last_change":"2024-03-26T05:21:34.850668+0000","last_active":"2024-03-26T05:22:45.889827+0000","last_peered":"2024-03-26T05:22:45.889827+0000","last_clean":"2024-03-26T05:22:45.889827+0000","last_became_active":"2024-03-26T05:21:34.850210+0000","last_became_peered":"2024-03-26T05:21:34.850210+0000","last_unstale":"2024-03-26T05:22:45.889827+0000","last_undegraded":"2024-03-26T05:22:45.889827+0000","last_fullsized":"2024-03-26T05:22:45.889827+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:50:15.781237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'2","reported_seq":40,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990670+0000","last_change":"2024-03-26T05:21:30.825374+0000","last_active":"2024-03-26T05:22:09.990670+0000","last_peered":"2024-03-26T05:22:09.990670+0000","last_clean":"2024-03-26T05:22:09.990670+0000","last_became_active":"2024-03-26T05:21:30.825023+0000","last_became_peered":"2024-03-26T05:21:30.825023+0000","last_unstale":"2024-03-26T05:22:09.990670+0000","last_undegraded":"2024-03-26T05:22:09.990670+0000","last_fullsized":"2024-03-26T05:22:09.990670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:20:41.536229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'16","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992595+0000","last_change":"2024-03-26T05:21:33.318076+0000","last_active":"2024-03-26T05:22:09.992595+0000","last_peered":"2024-03-26T05:22:09.992595+0000","last_clean":"2024-03-26T05:22:09.992595+0000","last_became_active":"2024-03-26T05:21:33.317781+0000","last_became_peered":"2024-03-26T05:21:33.317781+0000","last_unstale":"2024-03-26T05:22:09.992595+0000","last_undegraded":"2024-03-26T05:22:09.992595+0000","last_fullsized":"2024-03-26T05:22:09.992595+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:04:28.183791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991268+0000","last_change":"2024-03-26T05:21:36.857211+0000","last_active":"2024-03-26T05:22:09.991268+0000","last_peered":"2024-03-26T05:22:09.991268+0000","last_clean":"2024-03-26T05:22:09.991268+0000","last_became_active":"2024-03-26T05:21:36.856677+0000","last_became_peered":"2024-03-26T05:21:36.856677+0000","last_unstale":"2024-03-26T05:22:09.991268+0000","last_undegraded":"2024-03-26T05:22:09.991268+0000","last_fullsized":"2024-03-26T05:22:09.991268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:19:12.371760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992920+0000","last_change":"2024-03-26T05:21:34.855607+0000","last_active":"2024-03-26T05:22:09.992920+0000","last_peered":"2024-03-26T05:22:09.992920+0000","last_clean":"2024-03-26T05:22:09.992920+0000","last_became_active":"2024-03-26T05:21:34.855231+0000","last_became_peered":"2024-03-26T05:21:34.855231+0000","last_unstale":"2024-03-26T05:22:09.992920+0000","last_undegraded":"2024-03-26T05:22:09.992920+0000","last_fullsized":"2024-03-26T05:22:09.992920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:53.782244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807424+0000","last_change":"2024-03-26T05:21:30.826683+0000","last_active":"2024-03-26T05:22:09.807424+0000","last_peered":"2024-03-26T05:22:09.807424+0000","last_clean":"2024-03-26T05:22:09.807424+0000","last_became_active":"2024-03-26T05:21:30.826119+0000","last_became_peered":"2024-03-26T05:21:30.826119+0000","last_unstale":"2024-03-26T05:22:09.807424+0000","last_undegraded":"2024-03-26T05:22:09.807424+0000","last_fullsized":"2024-03-26T05:22:09.807424+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:42:31.524212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990761+0000","last_change":"2024-03-26T05:21:32.849949+0000","last_active":"2024-03-26T05:22:09.990761+0000","last_peered":"2024-03-26T05:22:09.990761+0000","last_clean":"2024-03-26T05:22:09.990761+0000","last_became_active":"2024-03-26T05:21:32.849625+0000","last_became_peered":"2024-03-26T05:21:32.849625+0000","last_unstale":"2024-03-26T05:22:09.990761+0000","last_undegraded":"2024-03-26T05:22:09.990761+0000","last_fullsized":"2024-03-26T05:22:09.990761+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:26:19.461625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923025+0000","last_change":"2024-03-26T05:21:36.857211+0000","last_active":"2024-03-26T05:22:09.923025+0000","last_peered":"2024-03-26T05:22:09.923025+0000","last_clean":"2024-03-26T05:22:09.923025+0000","last_became_active":"2024-03-26T05:21:36.856918+0000","last_became_peered":"2024-03-26T05:21:36.856918+0000","last_unstale":"2024-03-26T05:22:09.923025+0000","last_undegraded":"2024-03-26T05:22:09.923025+0000","last_fullsized":"2024-03-26T05:22:09.923025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:18:17.199617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990271+0000","last_change":"2024-03-26T05:21:34.848601+0000","last_active":"2024-03-26T05:22:09.990271+0000","last_peered":"2024-03-26T05:22:09.990271+0000","last_clean":"2024-03-26T05:22:09.990271+0000","last_became_active":"2024-03-26T05:21:34.848219+0000","last_became_peered":"2024-03-26T05:21:34.848219+0000","last_unstale":"2024-03-26T05:22:09.990271+0000","last_undegraded":"2024-03-26T05:22:09.990271+0000","last_fullsized":"2024-03-26T05:22:09.990271+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:19:07.038676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992141+0000","last_change":"2024-03-26T05:21:30.826495+0000","last_active":"2024-03-26T05:22:09.992141+0000","last_peered":"2024-03-26T05:22:09.992141+0000","last_clean":"2024-03-26T05:22:09.992141+0000","last_became_active":"2024-03-26T05:21:30.825954+0000","last_became_peered":"2024-03-26T05:21:30.825954+0000","last_unstale":"2024-03-26T05:22:09.992141+0000","last_undegraded":"2024-03-26T05:22:09.992141+0000","last_fullsized":"2024-03-26T05:22:09.992141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:32:24.921786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809973+0000","last_change":"2024-03-26T05:21:32.851635+0000","last_active":"2024-03-26T05:22:09.809973+0000","last_peered":"2024-03-26T05:22:09.809973+0000","last_clean":"2024-03-26T05:22:09.809973+0000","last_became_active":"2024-03-26T05:21:32.851323+0000","last_became_peered":"2024-03-26T05:21:32.851323+0000","last_unstale":"2024-03-26T05:22:09.809973+0000","last_undegraded":"2024-03-26T05:22:09.809973+0000","last_fullsized":"2024-03-26T05:22:09.809973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:05:50.958504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806352+0000","last_change":"2024-03-26T05:21:37.150189+0000","last_active":"2024-03-26T05:22:09.806352+0000","last_peered":"2024-03-26T05:22:09.806352+0000","last_clean":"2024-03-26T05:22:09.806352+0000","last_became_active":"2024-03-26T05:21:37.149777+0000","last_became_peered":"2024-03-26T05:21:37.149777+0000","last_unstale":"2024-03-26T05:22:09.806352+0000","last_undegraded":"2024-03-26T05:22:09.806352+0000","last_fullsized":"2024-03-26T05:22:09.806352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:10:40.698869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923196+0000","last_change":"2024-03-26T05:21:34.848587+0000","last_active":"2024-03-26T05:22:09.923196+0000","last_peered":"2024-03-26T05:22:09.923196+0000","last_clean":"2024-03-26T05:22:09.923196+0000","last_became_active":"2024-03-26T05:21:34.848279+0000","last_became_peered":"2024-03-26T05:21:34.848279+0000","last_unstale":"2024-03-26T05:22:09.923196+0000","last_undegraded":"2024-03-26T05:22:09.923196+0000","last_fullsized":"2024-03-26T05:22:09.923196+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:22:37.469760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912177+0000","last_change":"2024-03-26T05:21:30.824844+0000","last_active":"2024-03-26T05:22:09.912177+0000","last_peered":"2024-03-26T05:22:09.912177+0000","last_clean":"2024-03-26T05:22:09.912177+0000","last_became_active":"2024-03-26T05:21:30.824528+0000","last_became_peered":"2024-03-26T05:21:30.824528+0000","last_unstale":"2024-03-26T05:22:09.912177+0000","last_undegraded":"2024-03-26T05:22:09.912177+0000","last_fullsized":"2024-03-26T05:22:09.912177+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:34:03.013425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806738+0000","last_change":"2024-03-26T05:21:33.315420+0000","last_active":"2024-03-26T05:22:09.806738+0000","last_peered":"2024-03-26T05:22:09.806738+0000","last_clean":"2024-03-26T05:22:09.806738+0000","last_became_active":"2024-03-26T05:21:33.315099+0000","last_became_peered":"2024-03-26T05:21:33.315099+0000","last_unstale":"2024-03-26T05:22:09.806738+0000","last_undegraded":"2024-03-26T05:22:09.806738+0000","last_fullsized":"2024-03-26T05:22:09.806738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:04:06.283537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990334+0000","last_change":"2024-03-26T05:21:36.856760+0000","last_active":"2024-03-26T05:22:09.990334+0000","last_peered":"2024-03-26T05:22:09.990334+0000","last_clean":"2024-03-26T05:22:09.990334+0000","last_became_active":"2024-03-26T05:21:36.856457+0000","last_became_peered":"2024-03-26T05:21:36.856457+0000","last_unstale":"2024-03-26T05:22:09.990334+0000","last_undegraded":"2024-03-26T05:22:09.990334+0000","last_fullsized":"2024-03-26T05:22:09.990334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:56:41.932698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810368+0000","last_change":"2024-03-26T05:21:34.857757+0000","last_active":"2024-03-26T05:22:09.810368+0000","last_peered":"2024-03-26T05:22:09.810368+0000","last_clean":"2024-03-26T05:22:09.810368+0000","last_became_active":"2024-03-26T05:21:34.857396+0000","last_became_peered":"2024-03-26T05:21:34.857396+0000","last_unstale":"2024-03-26T05:22:09.810368+0000","last_undegraded":"2024-03-26T05:22:09.810368+0000","last_fullsized":"2024-03-26T05:22:09.810368+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:12.424403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923557+0000","last_change":"2024-03-26T05:21:30.833235+0000","last_active":"2024-03-26T05:22:09.923557+0000","last_peered":"2024-03-26T05:22:09.923557+0000","last_clean":"2024-03-26T05:22:09.923557+0000","last_became_active":"2024-03-26T05:21:30.832498+0000","last_became_peered":"2024-03-26T05:21:30.832498+0000","last_unstale":"2024-03-26T05:22:09.923557+0000","last_undegraded":"2024-03-26T05:22:09.923557+0000","last_fullsized":"2024-03-26T05:22:09.923557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:59:17.855155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991170+0000","last_change":"2024-03-26T05:21:32.847793+0000","last_active":"2024-03-26T05:22:09.991170+0000","last_peered":"2024-03-26T05:22:09.991170+0000","last_clean":"2024-03-26T05:22:09.991170+0000","last_became_active":"2024-03-26T05:21:32.847468+0000","last_became_peered":"2024-03-26T05:21:32.847468+0000","last_unstale":"2024-03-26T05:22:09.991170+0000","last_undegraded":"2024-03-26T05:22:09.991170+0000","last_fullsized":"2024-03-26T05:22:09.991170+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:26:47.840583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806414+0000","last_change":"2024-03-26T05:21:36.857092+0000","last_active":"2024-03-26T05:22:09.806414+0000","last_peered":"2024-03-26T05:22:09.806414+0000","last_clean":"2024-03-26T05:22:09.806414+0000","last_became_active":"2024-03-26T05:21:36.856471+0000","last_became_peered":"2024-03-26T05:21:36.856471+0000","last_unstale":"2024-03-26T05:22:09.806414+0000","last_undegraded":"2024-03-26T05:22:09.806414+0000","last_fullsized":"2024-03-26T05:22:09.806414+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:52:34.569101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990900+0000","last_change":"2024-03-26T05:21:34.850613+0000","last_active":"2024-03-26T05:22:09.990900+0000","last_peered":"2024-03-26T05:22:09.990900+0000","last_clean":"2024-03-26T05:22:09.990900+0000","last_became_active":"2024-03-26T05:21:34.849544+0000","last_became_peered":"2024-03-26T05:21:34.849544+0000","last_unstale":"2024-03-26T05:22:09.990900+0000","last_undegraded":"2024-03-26T05:22:09.990900+0000","last_fullsized":"2024-03-26T05:22:09.990900+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:47:45.134687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809857+0000","last_change":"2024-03-26T05:21:30.830415+0000","last_active":"2024-03-26T05:22:09.809857+0000","last_peered":"2024-03-26T05:22:09.809857+0000","last_clean":"2024-03-26T05:22:09.809857+0000","last_became_active":"2024-03-26T05:21:30.830091+0000","last_became_peered":"2024-03-26T05:21:30.830091+0000","last_unstale":"2024-03-26T05:22:09.809857+0000","last_undegraded":"2024-03-26T05:22:09.809857+0000","last_fullsized":"2024-03-26T05:22:09.809857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:06:33.998542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810699+0000","last_change":"2024-03-26T05:21:32.850623+0000","last_active":"2024-03-26T05:22:09.810699+0000","last_peered":"2024-03-26T05:22:09.810699+0000","last_clean":"2024-03-26T05:22:09.810699+0000","last_became_active":"2024-03-26T05:21:32.848371+0000","last_became_peered":"2024-03-26T05:21:32.848371+0000","last_unstale":"2024-03-26T05:22:09.810699+0000","last_undegraded":"2024-03-26T05:22:09.810699+0000","last_fullsized":"2024-03-26T05:22:09.810699+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:45:10.703984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992259+0000","last_change":"2024-03-26T05:21:36.859073+0000","last_active":"2024-03-26T05:22:09.992259+0000","last_peered":"2024-03-26T05:22:09.992259+0000","last_clean":"2024-03-26T05:22:09.992259+0000","last_became_active":"2024-03-26T05:21:36.858609+0000","last_became_peered":"2024-03-26T05:21:36.858609+0000","last_unstale":"2024-03-26T05:22:09.992259+0000","last_undegraded":"2024-03-26T05:22:09.992259+0000","last_fullsized":"2024-03-26T05:22:09.992259+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:18.608563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889826+0000","last_change":"2024-03-26T05:21:34.853358+0000","last_active":"2024-03-26T05:22:45.889826+0000","last_peered":"2024-03-26T05:22:45.889826+0000","last_clean":"2024-03-26T05:22:45.889826+0000","last_became_active":"2024-03-26T05:21:34.853068+0000","last_became_peered":"2024-03-26T05:21:34.853068+0000","last_unstale":"2024-03-26T05:22:45.889826+0000","last_undegraded":"2024-03-26T05:22:45.889826+0000","last_fullsized":"2024-03-26T05:22:45.889826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:00:31.335053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923166+0000","last_change":"2024-03-26T05:21:30.835120+0000","last_active":"2024-03-26T05:22:09.923166+0000","last_peered":"2024-03-26T05:22:09.923166+0000","last_clean":"2024-03-26T05:22:09.923166+0000","last_became_active":"2024-03-26T05:21:30.834834+0000","last_became_peered":"2024-03-26T05:21:30.834834+0000","last_unstale":"2024-03-26T05:22:09.923166+0000","last_undegraded":"2024-03-26T05:22:09.923166+0000","last_fullsized":"2024-03-26T05:22:09.923166+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:31:02.545407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'22","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807147+0000","last_change":"2024-03-26T05:21:33.316317+0000","last_active":"2024-03-26T05:22:09.807147+0000","last_peered":"2024-03-26T05:22:09.807147+0000","last_clean":"2024-03-26T05:22:09.807147+0000","last_became_active":"2024-03-26T05:21:33.316042+0000","last_became_peered":"2024-03-26T05:21:33.316042+0000","last_unstale":"2024-03-26T05:22:09.807147+0000","last_undegraded":"2024-03-26T05:22:09.807147+0000","last_fullsized":"2024-03-26T05:22:09.807147+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:10:12.931440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810349+0000","last_change":"2024-03-26T05:21:36.856309+0000","last_active":"2024-03-26T05:22:09.810349+0000","last_peered":"2024-03-26T05:22:09.810349+0000","last_clean":"2024-03-26T05:22:09.810349+0000","last_became_active":"2024-03-26T05:21:36.856030+0000","last_became_peered":"2024-03-26T05:21:36.856030+0000","last_unstale":"2024-03-26T05:22:09.810349+0000","last_undegraded":"2024-03-26T05:22:09.810349+0000","last_fullsized":"2024-03-26T05:22:09.810349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:20:36.064630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889507+0000","last_change":"2024-03-26T05:21:34.847671+0000","last_active":"2024-03-26T05:22:45.889507+0000","last_peered":"2024-03-26T05:22:45.889507+0000","last_clean":"2024-03-26T05:22:45.889507+0000","last_became_active":"2024-03-26T05:21:34.847169+0000","last_became_peered":"2024-03-26T05:21:34.847169+0000","last_unstale":"2024-03-26T05:22:45.889507+0000","last_undegraded":"2024-03-26T05:22:45.889507+0000","last_fullsized":"2024-03-26T05:22:45.889507+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:21:51.596952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991792+0000","last_change":"2024-03-26T05:21:30.827607+0000","last_active":"2024-03-26T05:22:09.991792+0000","last_peered":"2024-03-26T05:22:09.991792+0000","last_clean":"2024-03-26T05:22:09.991792+0000","last_became_active":"2024-03-26T05:21:30.827141+0000","last_became_peered":"2024-03-26T05:21:30.827141+0000","last_unstale":"2024-03-26T05:22:09.991792+0000","last_undegraded":"2024-03-26T05:22:09.991792+0000","last_fullsized":"2024-03-26T05:22:09.991792+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:25:39.858325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'15","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912629+0000","last_change":"2024-03-26T05:21:33.316834+0000","last_active":"2024-03-26T05:22:09.912629+0000","last_peered":"2024-03-26T05:22:09.912629+0000","last_clean":"2024-03-26T05:22:09.912629+0000","last_became_active":"2024-03-26T05:21:33.316512+0000","last_became_peered":"2024-03-26T05:21:33.316512+0000","last_unstale":"2024-03-26T05:22:09.912629+0000","last_undegraded":"2024-03-26T05:22:09.912629+0000","last_fullsized":"2024-03-26T05:22:09.912629+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:27:01.739178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807068+0000","last_change":"2024-03-26T05:21:36.859896+0000","last_active":"2024-03-26T05:22:09.807068+0000","last_peered":"2024-03-26T05:22:09.807068+0000","last_clean":"2024-03-26T05:22:09.807068+0000","last_became_active":"2024-03-26T05:21:36.859613+0000","last_became_peered":"2024-03-26T05:21:36.859613+0000","last_unstale":"2024-03-26T05:22:09.807068+0000","last_undegraded":"2024-03-26T05:22:09.807068+0000","last_fullsized":"2024-03-26T05:22:09.807068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:56:58.193646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808639+0000","last_change":"2024-03-26T05:21:34.849127+0000","last_active":"2024-03-26T05:22:09.808639+0000","last_peered":"2024-03-26T05:22:09.808639+0000","last_clean":"2024-03-26T05:22:09.808639+0000","last_became_active":"2024-03-26T05:21:34.848831+0000","last_became_peered":"2024-03-26T05:21:34.848831+0000","last_unstale":"2024-03-26T05:22:09.808639+0000","last_undegraded":"2024-03-26T05:22:09.808639+0000","last_fullsized":"2024-03-26T05:22:09.808639+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:41:33.435188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923106+0000","last_change":"2024-03-26T05:21:30.831882+0000","last_active":"2024-03-26T05:22:09.923106+0000","last_peered":"2024-03-26T05:22:09.923106+0000","last_clean":"2024-03-26T05:22:09.923106+0000","last_became_active":"2024-03-26T05:21:30.831563+0000","last_became_peered":"2024-03-26T05:21:30.831563+0000","last_unstale":"2024-03-26T05:22:09.923106+0000","last_undegraded":"2024-03-26T05:22:09.923106+0000","last_fullsized":"2024-03-26T05:22:09.923106+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:55:51.245612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'23","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992726+0000","last_change":"2024-03-26T05:21:33.315441+0000","last_active":"2024-03-26T05:22:09.992726+0000","last_peered":"2024-03-26T05:22:09.992726+0000","last_clean":"2024-03-26T05:22:09.992726+0000","last_became_active":"2024-03-26T05:21:33.315118+0000","last_became_peered":"2024-03-26T05:21:33.315118+0000","last_unstale":"2024-03-26T05:22:09.992726+0000","last_undegraded":"2024-03-26T05:22:09.992726+0000","last_fullsized":"2024-03-26T05:22:09.992726+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:08:35.526716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923462+0000","last_change":"2024-03-26T05:21:37.150099+0000","last_active":"2024-03-26T05:22:09.923462+0000","last_peered":"2024-03-26T05:22:09.923462+0000","last_clean":"2024-03-26T05:22:09.923462+0000","last_became_active":"2024-03-26T05:21:37.149768+0000","last_became_peered":"2024-03-26T05:21:37.149768+0000","last_unstale":"2024-03-26T05:22:09.923462+0000","last_undegraded":"2024-03-26T05:22:09.923462+0000","last_fullsized":"2024-03-26T05:22:09.923462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:35:45.876849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912517+0000","last_change":"2024-03-26T05:21:34.852183+0000","last_active":"2024-03-26T05:22:09.912517+0000","last_peered":"2024-03-26T05:22:09.912517+0000","last_clean":"2024-03-26T05:22:09.912517+0000","last_became_active":"2024-03-26T05:21:34.851872+0000","last_became_peered":"2024-03-26T05:21:34.851872+0000","last_unstale":"2024-03-26T05:22:09.912517+0000","last_undegraded":"2024-03-26T05:22:09.912517+0000","last_fullsized":"2024-03-26T05:22:09.912517+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:23:09.538518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990284+0000","last_change":"2024-03-26T05:21:30.826246+0000","last_active":"2024-03-26T05:22:09.990284+0000","last_peered":"2024-03-26T05:22:09.990284+0000","last_clean":"2024-03-26T05:22:09.990284+0000","last_became_active":"2024-03-26T05:21:30.825958+0000","last_became_peered":"2024-03-26T05:21:30.825958+0000","last_unstale":"2024-03-26T05:22:09.990284+0000","last_undegraded":"2024-03-26T05:22:09.990284+0000","last_fullsized":"2024-03-26T05:22:09.990284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:18.090816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'12","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808820+0000","last_change":"2024-03-26T05:21:33.316854+0000","last_active":"2024-03-26T05:22:09.808820+0000","last_peered":"2024-03-26T05:22:09.808820+0000","last_clean":"2024-03-26T05:22:09.808820+0000","last_became_active":"2024-03-26T05:21:33.316574+0000","last_became_peered":"2024-03-26T05:21:33.316574+0000","last_unstale":"2024-03-26T05:22:09.808820+0000","last_undegraded":"2024-03-26T05:22:09.808820+0000","last_fullsized":"2024-03-26T05:22:09.808820+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:49:25.559854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911922+0000","last_change":"2024-03-26T05:21:37.148965+0000","last_active":"2024-03-26T05:22:09.911922+0000","last_peered":"2024-03-26T05:22:09.911922+0000","last_clean":"2024-03-26T05:22:09.911922+0000","last_became_active":"2024-03-26T05:21:37.147966+0000","last_became_peered":"2024-03-26T05:21:37.147966+0000","last_unstale":"2024-03-26T05:22:09.911922+0000","last_undegraded":"2024-03-26T05:22:09.911922+0000","last_fullsized":"2024-03-26T05:22:09.911922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:47:52.078291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923930+0000","last_change":"2024-03-26T05:21:30.835220+0000","last_active":"2024-03-26T05:22:09.923930+0000","last_peered":"2024-03-26T05:22:09.923930+0000","last_clean":"2024-03-26T05:22:09.923930+0000","last_became_active":"2024-03-26T05:21:30.834868+0000","last_became_peered":"2024-03-26T05:21:30.834868+0000","last_unstale":"2024-03-26T05:22:09.923930+0000","last_undegraded":"2024-03-26T05:22:09.923930+0000","last_fullsized":"2024-03-26T05:22:09.923930+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:38:22.083221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'27","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911548+0000","last_change":"2024-03-26T05:21:33.316316+0000","last_active":"2024-03-26T05:22:09.911548+0000","last_peered":"2024-03-26T05:22:09.911548+0000","last_clean":"2024-03-26T05:22:09.911548+0000","last_became_active":"2024-03-26T05:21:33.315943+0000","last_became_peered":"2024-03-26T05:21:33.315943+0000","last_unstale":"2024-03-26T05:22:09.911548+0000","last_undegraded":"2024-03-26T05:22:09.911548+0000","last_fullsized":"2024-03-26T05:22:09.911548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:56:04.964323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889821+0000","last_change":"2024-03-26T05:21:34.849489+0000","last_active":"2024-03-26T05:22:45.889821+0000","last_peered":"2024-03-26T05:22:45.889821+0000","last_clean":"2024-03-26T05:22:45.889821+0000","last_became_active":"2024-03-26T05:21:34.849194+0000","last_became_peered":"2024-03-26T05:21:34.849194+0000","last_unstale":"2024-03-26T05:22:45.889821+0000","last_undegraded":"2024-03-26T05:22:45.889821+0000","last_fullsized":"2024-03-26T05:22:45.889821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:52.786092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807295+0000","last_change":"2024-03-26T05:21:36.853858+0000","last_active":"2024-03-26T05:22:09.807295+0000","last_peered":"2024-03-26T05:22:09.807295+0000","last_clean":"2024-03-26T05:22:09.807295+0000","last_became_active":"2024-03-26T05:21:36.853556+0000","last_became_peered":"2024-03-26T05:21:36.853556+0000","last_unstale":"2024-03-26T05:22:09.807295+0000","last_undegraded":"2024-03-26T05:22:09.807295+0000","last_fullsized":"2024-03-26T05:22:09.807295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:00:51.819093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923805+0000","last_change":"2024-03-26T05:21:30.834326+0000","last_active":"2024-03-26T05:22:09.923805+0000","last_peered":"2024-03-26T05:22:09.923805+0000","last_clean":"2024-03-26T05:22:09.923805+0000","last_became_active":"2024-03-26T05:21:30.833967+0000","last_became_peered":"2024-03-26T05:21:30.833967+0000","last_unstale":"2024-03-26T05:22:09.923805+0000","last_undegraded":"2024-03-26T05:22:09.923805+0000","last_fullsized":"2024-03-26T05:22:09.923805+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:18:03.676676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'24","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809274+0000","last_change":"2024-03-26T05:21:32.850489+0000","last_active":"2024-03-26T05:22:09.809274+0000","last_peered":"2024-03-26T05:22:09.809274+0000","last_clean":"2024-03-26T05:22:09.809274+0000","last_became_active":"2024-03-26T05:21:32.850120+0000","last_became_peered":"2024-03-26T05:21:32.850120+0000","last_unstale":"2024-03-26T05:22:09.809274+0000","last_undegraded":"2024-03-26T05:22:09.809274+0000","last_fullsized":"2024-03-26T05:22:09.809274+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:23:04.138636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.805298+0000","last_change":"2024-03-26T05:21:34.854951+0000","last_active":"2024-03-26T05:22:09.805298+0000","last_peered":"2024-03-26T05:22:09.805298+0000","last_clean":"2024-03-26T05:22:09.805298+0000","last_became_active":"2024-03-26T05:21:34.854626+0000","last_became_peered":"2024-03-26T05:21:34.854626+0000","last_unstale":"2024-03-26T05:22:09.805298+0000","last_undegraded":"2024-03-26T05:22:09.805298+0000","last_fullsized":"2024-03-26T05:22:09.805298+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:14:39.612969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.922880+0000","last_change":"2024-03-26T05:21:36.856466+0000","last_active":"2024-03-26T05:22:09.922880+0000","last_peered":"2024-03-26T05:22:09.922880+0000","last_clean":"2024-03-26T05:22:09.922880+0000","last_became_active":"2024-03-26T05:21:36.856184+0000","last_became_peered":"2024-03-26T05:21:36.856184+0000","last_unstale":"2024-03-26T05:22:09.922880+0000","last_undegraded":"2024-03-26T05:22:09.922880+0000","last_fullsized":"2024-03-26T05:22:09.922880+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:14:33.325922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806285+0000","last_change":"2024-03-26T05:21:34.853291+0000","last_active":"2024-03-26T05:22:09.806285+0000","last_peered":"2024-03-26T05:22:09.806285+0000","last_clean":"2024-03-26T05:22:09.806285+0000","last_became_active":"2024-03-26T05:21:34.852985+0000","last_became_peered":"2024-03-26T05:21:34.852985+0000","last_unstale":"2024-03-26T05:22:09.806285+0000","last_undegraded":"2024-03-26T05:22:09.806285+0000","last_fullsized":"2024-03-26T05:22:09.806285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:27:32.396824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923671+0000","last_change":"2024-03-26T05:21:30.829712+0000","last_active":"2024-03-26T05:22:09.923671+0000","last_peered":"2024-03-26T05:22:09.923671+0000","last_clean":"2024-03-26T05:22:09.923671+0000","last_became_active":"2024-03-26T05:21:30.829418+0000","last_became_peered":"2024-03-26T05:21:30.829418+0000","last_unstale":"2024-03-26T05:22:09.923671+0000","last_undegraded":"2024-03-26T05:22:09.923671+0000","last_fullsized":"2024-03-26T05:22:09.923671+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:02:34.857021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'25","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990802+0000","last_change":"2024-03-26T05:21:33.318417+0000","last_active":"2024-03-26T05:22:09.990802+0000","last_peered":"2024-03-26T05:22:09.990802+0000","last_clean":"2024-03-26T05:22:09.990802+0000","last_became_active":"2024-03-26T05:21:33.317980+0000","last_became_peered":"2024-03-26T05:21:33.317980+0000","last_unstale":"2024-03-26T05:22:09.990802+0000","last_undegraded":"2024-03-26T05:22:09.990802+0000","last_fullsized":"2024-03-26T05:22:09.990802+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:14:57.144778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991684+0000","last_change":"2024-03-26T05:21:36.858892+0000","last_active":"2024-03-26T05:22:09.991684+0000","last_peered":"2024-03-26T05:22:09.991684+0000","last_clean":"2024-03-26T05:22:09.991684+0000","last_became_active":"2024-03-26T05:21:36.858505+0000","last_became_peered":"2024-03-26T05:21:36.858505+0000","last_unstale":"2024-03-26T05:22:09.991684+0000","last_undegraded":"2024-03-26T05:22:09.991684+0000","last_fullsized":"2024-03-26T05:22:09.991684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:39:08.552790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.805938+0000","last_change":"2024-03-26T05:21:34.842418+0000","last_active":"2024-03-26T05:22:09.805938+0000","last_peered":"2024-03-26T05:22:09.805938+0000","last_clean":"2024-03-26T05:22:09.805938+0000","last_became_active":"2024-03-26T05:21:34.842126+0000","last_became_peered":"2024-03-26T05:21:34.842126+0000","last_unstale":"2024-03-26T05:22:09.805938+0000","last_undegraded":"2024-03-26T05:22:09.805938+0000","last_fullsized":"2024-03-26T05:22:09.805938+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:18:10.305337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911963+0000","last_change":"2024-03-26T05:21:30.826423+0000","last_active":"2024-03-26T05:22:09.911963+0000","last_peered":"2024-03-26T05:22:09.911963+0000","last_clean":"2024-03-26T05:22:09.911963+0000","last_became_active":"2024-03-26T05:21:30.826094+0000","last_became_peered":"2024-03-26T05:21:30.826094+0000","last_unstale":"2024-03-26T05:22:09.911963+0000","last_undegraded":"2024-03-26T05:22:09.911963+0000","last_fullsized":"2024-03-26T05:22:09.911963+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:37:27.454663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'15","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992404+0000","last_change":"2024-03-26T05:21:57.699762+0000","last_active":"2024-03-26T05:22:09.992404+0000","last_peered":"2024-03-26T05:22:09.992404+0000","last_clean":"2024-03-26T05:22:09.992404+0000","last_became_active":"2024-03-26T05:21:57.687329+0000","last_became_peered":"2024-03-26T05:21:57.687329+0000","last_unstale":"2024-03-26T05:22:09.992404+0000","last_undegraded":"2024-03-26T05:22:09.992404+0000","last_fullsized":"2024-03-26T05:22:09.992404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:32:31.194632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.807076+0000","last_change":"2024-03-26T05:21:36.860966+0000","last_active":"2024-03-26T05:22:09.807076+0000","last_peered":"2024-03-26T05:22:09.807076+0000","last_clean":"2024-03-26T05:22:09.807076+0000","last_became_active":"2024-03-26T05:21:36.860629+0000","last_became_peered":"2024-03-26T05:21:36.860629+0000","last_unstale":"2024-03-26T05:22:09.807076+0000","last_undegraded":"2024-03-26T05:22:09.807076+0000","last_fullsized":"2024-03-26T05:22:09.807076+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:10:07.550878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911668+0000","last_change":"2024-03-26T05:21:33.839958+0000","last_active":"2024-03-26T05:22:09.911668+0000","last_peered":"2024-03-26T05:22:09.911668+0000","last_clean":"2024-03-26T05:22:09.911668+0000","last_became_active":"2024-03-26T05:21:29.817501+0000","last_became_peered":"2024-03-26T05:21:29.817501+0000","last_unstale":"2024-03-26T05:22:09.911668+0000","last_undegraded":"2024-03-26T05:22:09.911668+0000","last_fullsized":"2024-03-26T05:22:09.911668+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_clean_scrub_stamp":"2024-03-26T05:21:28.796453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:13:34.952002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052850800000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809189+0000","last_change":"2024-03-26T05:21:30.826660+0000","last_active":"2024-03-26T05:22:09.809189+0000","last_peered":"2024-03-26T05:22:09.809189+0000","last_clean":"2024-03-26T05:22:09.809189+0000","last_became_active":"2024-03-26T05:21:30.826338+0000","last_became_peered":"2024-03-26T05:21:30.826338+0000","last_unstale":"2024-03-26T05:22:09.809189+0000","last_undegraded":"2024-03-26T05:22:09.809189+0000","last_fullsized":"2024-03-26T05:22:09.809189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:05:54.055875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'34","reported_seq":106,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889437+0000","last_change":"2024-03-26T05:21:32.847971+0000","last_active":"2024-03-26T05:22:45.889437+0000","last_peered":"2024-03-26T05:22:45.889437+0000","last_clean":"2024-03-26T05:22:45.889437+0000","last_became_active":"2024-03-26T05:21:32.847624+0000","last_became_peered":"2024-03-26T05:21:32.847624+0000","last_unstale":"2024-03-26T05:22:45.889437+0000","last_undegraded":"2024-03-26T05:22:45.889437+0000","last_fullsized":"2024-03-26T05:22:45.889437+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:17:46.448596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992159+0000","last_change":"2024-03-26T05:21:34.852873+0000","last_active":"2024-03-26T05:22:09.992159+0000","last_peered":"2024-03-26T05:22:09.992159+0000","last_clean":"2024-03-26T05:22:09.992159+0000","last_became_active":"2024-03-26T05:21:34.852506+0000","last_became_peered":"2024-03-26T05:21:34.852506+0000","last_unstale":"2024-03-26T05:22:09.992159+0000","last_undegraded":"2024-03-26T05:22:09.992159+0000","last_fullsized":"2024-03-26T05:22:09.992159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:15:28.753927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923756+0000","last_change":"2024-03-26T05:21:37.150798+0000","last_active":"2024-03-26T05:22:09.923756+0000","last_peered":"2024-03-26T05:22:09.923756+0000","last_clean":"2024-03-26T05:22:09.923756+0000","last_became_active":"2024-03-26T05:21:37.150491+0000","last_became_peered":"2024-03-26T05:21:37.150491+0000","last_unstale":"2024-03-26T05:22:09.923756+0000","last_undegraded":"2024-03-26T05:22:09.923756+0000","last_fullsized":"2024-03-26T05:22:09.923756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:00.686155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990625+0000","last_change":"2024-03-26T05:21:30.829356+0000","last_active":"2024-03-26T05:22:09.990625+0000","last_peered":"2024-03-26T05:22:09.990625+0000","last_clean":"2024-03-26T05:22:09.990625+0000","last_became_active":"2024-03-26T05:21:30.829035+0000","last_became_peered":"2024-03-26T05:21:30.829035+0000","last_unstale":"2024-03-26T05:22:09.990625+0000","last_undegraded":"2024-03-26T05:22:09.990625+0000","last_fullsized":"2024-03-26T05:22:09.990625+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:56:02.276546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.805585+0000","last_change":"2024-03-26T05:21:33.835345+0000","last_active":"2024-03-26T05:22:09.805585+0000","last_peered":"2024-03-26T05:22:09.805585+0000","last_clean":"2024-03-26T05:22:09.805585+0000","last_became_active":"2024-03-26T05:21:29.813603+0000","last_became_peered":"2024-03-26T05:21:29.813603+0000","last_unstale":"2024-03-26T05:22:09.805585+0000","last_undegraded":"2024-03-26T05:22:09.805585+0000","last_fullsized":"2024-03-26T05:22:09.805585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_clean_scrub_stamp":"2024-03-26T05:21:28.796453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:31:14.841877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053699499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'19","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991908+0000","last_change":"2024-03-26T05:21:32.851263+0000","last_active":"2024-03-26T05:22:09.991908+0000","last_peered":"2024-03-26T05:22:09.991908+0000","last_clean":"2024-03-26T05:22:09.991908+0000","last_became_active":"2024-03-26T05:21:32.849895+0000","last_became_peered":"2024-03-26T05:21:32.849895+0000","last_unstale":"2024-03-26T05:22:09.991908+0000","last_undegraded":"2024-03-26T05:22:09.991908+0000","last_fullsized":"2024-03-26T05:22:09.991908+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:09:57.668003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.805353+0000","last_change":"2024-03-26T05:21:34.850606+0000","last_active":"2024-03-26T05:22:09.805353+0000","last_peered":"2024-03-26T05:22:09.805353+0000","last_clean":"2024-03-26T05:22:09.805353+0000","last_became_active":"2024-03-26T05:21:34.850299+0000","last_became_peered":"2024-03-26T05:21:34.850299+0000","last_unstale":"2024-03-26T05:22:09.805353+0000","last_undegraded":"2024-03-26T05:22:09.805353+0000","last_fullsized":"2024-03-26T05:22:09.805353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:57.004815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912186+0000","last_change":"2024-03-26T05:21:36.857575+0000","last_active":"2024-03-26T05:22:09.912186+0000","last_peered":"2024-03-26T05:22:09.912186+0000","last_clean":"2024-03-26T05:22:09.912186+0000","last_became_active":"2024-03-26T05:21:36.857203+0000","last_became_peered":"2024-03-26T05:21:36.857203+0000","last_unstale":"2024-03-26T05:22:09.912186+0000","last_undegraded":"2024-03-26T05:22:09.912186+0000","last_fullsized":"2024-03-26T05:22:09.912186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:05:37.683858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":278,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:11.901478+0000","last_change":"2024-03-26T05:21:21.195182+0000","last_active":"2024-03-26T05:22:11.901478+0000","last_peered":"2024-03-26T05:22:11.901478+0000","last_clean":"2024-03-26T05:22:11.901478+0000","last_became_active":"2024-03-26T05:21:21.194654+0000","last_became_peered":"2024-03-26T05:21:21.194654+0000","last_unstale":"2024-03-26T05:22:11.901478+0000","last_undegraded":"2024-03-26T05:22:11.901478+0000","last_fullsized":"2024-03-26T05:22:11.901478+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:19:15.266592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:19:15.266592+0000","last_clean_scrub_stamp":"2024-03-26T05:19:15.266592+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:47.295096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808019+0000","last_change":"2024-03-26T05:21:30.825786+0000","last_active":"2024-03-26T05:22:09.808019+0000","last_peered":"2024-03-26T05:22:09.808019+0000","last_clean":"2024-03-26T05:22:09.808019+0000","last_became_active":"2024-03-26T05:21:30.825516+0000","last_became_peered":"2024-03-26T05:21:30.825516+0000","last_unstale":"2024-03-26T05:22:09.808019+0000","last_undegraded":"2024-03-26T05:22:09.808019+0000","last_fullsized":"2024-03-26T05:22:09.808019+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:00:14.707568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.890108+0000","last_change":"2024-03-26T05:21:33.315830+0000","last_active":"2024-03-26T05:22:45.890108+0000","last_peered":"2024-03-26T05:22:45.890108+0000","last_clean":"2024-03-26T05:22:45.890108+0000","last_became_active":"2024-03-26T05:21:33.315413+0000","last_became_peered":"2024-03-26T05:21:33.315413+0000","last_unstale":"2024-03-26T05:22:45.890108+0000","last_undegraded":"2024-03-26T05:22:45.890108+0000","last_fullsized":"2024-03-26T05:22:45.890108+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:08:58.855063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923486+0000","last_change":"2024-03-26T05:21:34.853592+0000","last_active":"2024-03-26T05:22:09.923486+0000","last_peered":"2024-03-26T05:22:09.923486+0000","last_clean":"2024-03-26T05:22:09.923486+0000","last_became_active":"2024-03-26T05:21:34.852461+0000","last_became_peered":"2024-03-26T05:21:34.852461+0000","last_unstale":"2024-03-26T05:22:09.923486+0000","last_undegraded":"2024-03-26T05:22:09.923486+0000","last_fullsized":"2024-03-26T05:22:09.923486+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:55:28.147166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991003+0000","last_change":"2024-03-26T05:21:36.857365+0000","last_active":"2024-03-26T05:22:09.991003+0000","last_peered":"2024-03-26T05:22:09.991003+0000","last_clean":"2024-03-26T05:22:09.991003+0000","last_became_active":"2024-03-26T05:21:36.856242+0000","last_became_peered":"2024-03-26T05:21:36.856242+0000","last_unstale":"2024-03-26T05:22:09.991003+0000","last_undegraded":"2024-03-26T05:22:09.991003+0000","last_fullsized":"2024-03-26T05:22:09.991003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:15:05.564582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991925+0000","last_change":"2024-03-26T05:21:30.832001+0000","last_active":"2024-03-26T05:22:09.991925+0000","last_peered":"2024-03-26T05:22:09.991925+0000","last_clean":"2024-03-26T05:22:09.991925+0000","last_became_active":"2024-03-26T05:21:30.831107+0000","last_became_peered":"2024-03-26T05:21:30.831107+0000","last_unstale":"2024-03-26T05:22:09.991925+0000","last_undegraded":"2024-03-26T05:22:09.991925+0000","last_fullsized":"2024-03-26T05:22:09.991925+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:26:55.425511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:53.935153+0000","last_change":"2024-03-26T05:21:33.844834+0000","last_active":"2024-03-26T05:22:53.935153+0000","last_peered":"2024-03-26T05:22:53.935153+0000","last_clean":"2024-03-26T05:22:53.935153+0000","last_became_active":"2024-03-26T05:21:29.818246+0000","last_became_peered":"2024-03-26T05:21:29.818246+0000","last_unstale":"2024-03-26T05:22:53.935153+0000","last_undegraded":"2024-03-26T05:22:53.935153+0000","last_fullsized":"2024-03-26T05:22:53.935153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:28.796453+0000","last_clean_scrub_stamp":"2024-03-26T05:21:28.796453+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:34:55.722020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000459813,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992356+0000","last_change":"2024-03-26T05:21:32.848362+0000","last_active":"2024-03-26T05:22:09.992356+0000","last_peered":"2024-03-26T05:22:09.992356+0000","last_clean":"2024-03-26T05:22:09.992356+0000","last_became_active":"2024-03-26T05:21:32.848080+0000","last_became_peered":"2024-03-26T05:21:32.848080+0000","last_unstale":"2024-03-26T05:22:09.992356+0000","last_undegraded":"2024-03-26T05:22:09.992356+0000","last_fullsized":"2024-03-26T05:22:09.992356+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:59:10.757138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912618+0000","last_change":"2024-03-26T05:21:34.855388+0000","last_active":"2024-03-26T05:22:09.912618+0000","last_peered":"2024-03-26T05:22:09.912618+0000","last_clean":"2024-03-26T05:22:09.912618+0000","last_became_active":"2024-03-26T05:21:34.855073+0000","last_became_peered":"2024-03-26T05:21:34.855073+0000","last_unstale":"2024-03-26T05:22:09.912618+0000","last_undegraded":"2024-03-26T05:22:09.912618+0000","last_fullsized":"2024-03-26T05:22:09.912618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:48:15.631848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990457+0000","last_change":"2024-03-26T05:21:37.148788+0000","last_active":"2024-03-26T05:22:09.990457+0000","last_peered":"2024-03-26T05:22:09.990457+0000","last_clean":"2024-03-26T05:22:09.990457+0000","last_became_active":"2024-03-26T05:21:37.148328+0000","last_became_peered":"2024-03-26T05:21:37.148328+0000","last_unstale":"2024-03-26T05:22:09.990457+0000","last_undegraded":"2024-03-26T05:22:09.990457+0000","last_fullsized":"2024-03-26T05:22:09.990457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:34:21.274218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991449+0000","last_change":"2024-03-26T05:21:30.824900+0000","last_active":"2024-03-26T05:22:09.991449+0000","last_peered":"2024-03-26T05:22:09.991449+0000","last_clean":"2024-03-26T05:22:09.991449+0000","last_became_active":"2024-03-26T05:21:30.824505+0000","last_became_peered":"2024-03-26T05:21:30.824505+0000","last_unstale":"2024-03-26T05:22:09.991449+0000","last_undegraded":"2024-03-26T05:22:09.991449+0000","last_fullsized":"2024-03-26T05:22:09.991449+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:10:01.721083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'26","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992737+0000","last_change":"2024-03-26T05:21:33.315461+0000","last_active":"2024-03-26T05:22:09.992737+0000","last_peered":"2024-03-26T05:22:09.992737+0000","last_clean":"2024-03-26T05:22:09.992737+0000","last_became_active":"2024-03-26T05:21:33.315097+0000","last_became_peered":"2024-03-26T05:21:33.315097+0000","last_unstale":"2024-03-26T05:22:09.992737+0000","last_undegraded":"2024-03-26T05:22:09.992737+0000","last_fullsized":"2024-03-26T05:22:09.992737+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:48:50.911641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990804+0000","last_change":"2024-03-26T05:21:34.852403+0000","last_active":"2024-03-26T05:22:09.990804+0000","last_peered":"2024-03-26T05:22:09.990804+0000","last_clean":"2024-03-26T05:22:09.990804+0000","last_became_active":"2024-03-26T05:21:34.852110+0000","last_became_peered":"2024-03-26T05:21:34.852110+0000","last_unstale":"2024-03-26T05:22:09.990804+0000","last_undegraded":"2024-03-26T05:22:09.990804+0000","last_fullsized":"2024-03-26T05:22:09.990804+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:42:10.729353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991074+0000","last_change":"2024-03-26T05:21:36.856082+0000","last_active":"2024-03-26T05:22:09.991074+0000","last_peered":"2024-03-26T05:22:09.991074+0000","last_clean":"2024-03-26T05:22:09.991074+0000","last_became_active":"2024-03-26T05:21:36.855787+0000","last_became_peered":"2024-03-26T05:21:36.855787+0000","last_unstale":"2024-03-26T05:22:09.991074+0000","last_undegraded":"2024-03-26T05:22:09.991074+0000","last_fullsized":"2024-03-26T05:22:09.991074+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:08:46.655147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912215+0000","last_change":"2024-03-26T05:21:30.819467+0000","last_active":"2024-03-26T05:22:09.912215+0000","last_peered":"2024-03-26T05:22:09.912215+0000","last_clean":"2024-03-26T05:22:09.912215+0000","last_became_active":"2024-03-26T05:21:30.819175+0000","last_became_peered":"2024-03-26T05:21:30.819175+0000","last_unstale":"2024-03-26T05:22:09.912215+0000","last_undegraded":"2024-03-26T05:22:09.912215+0000","last_fullsized":"2024-03-26T05:22:09.912215+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:04:11.909175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'16","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991021+0000","last_change":"2024-03-26T05:21:32.848936+0000","last_active":"2024-03-26T05:22:09.991021+0000","last_peered":"2024-03-26T05:22:09.991021+0000","last_clean":"2024-03-26T05:22:09.991021+0000","last_became_active":"2024-03-26T05:21:32.848369+0000","last_became_peered":"2024-03-26T05:21:32.848369+0000","last_unstale":"2024-03-26T05:22:09.991021+0000","last_undegraded":"2024-03-26T05:22:09.991021+0000","last_fullsized":"2024-03-26T05:22:09.991021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:04:00.986624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889450+0000","last_change":"2024-03-26T05:21:34.852447+0000","last_active":"2024-03-26T05:22:45.889450+0000","last_peered":"2024-03-26T05:22:45.889450+0000","last_clean":"2024-03-26T05:22:45.889450+0000","last_became_active":"2024-03-26T05:21:34.852034+0000","last_became_peered":"2024-03-26T05:21:34.852034+0000","last_unstale":"2024-03-26T05:22:45.889450+0000","last_undegraded":"2024-03-26T05:22:45.889450+0000","last_fullsized":"2024-03-26T05:22:45.889450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:36:08.950749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.923397+0000","last_change":"2024-03-26T05:21:37.150760+0000","last_active":"2024-03-26T05:22:09.923397+0000","last_peered":"2024-03-26T05:22:09.923397+0000","last_clean":"2024-03-26T05:22:09.923397+0000","last_became_active":"2024-03-26T05:21:37.150451+0000","last_became_peered":"2024-03-26T05:21:37.150451+0000","last_unstale":"2024-03-26T05:22:09.923397+0000","last_undegraded":"2024-03-26T05:22:09.923397+0000","last_fullsized":"2024-03-26T05:22:09.923397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:50:55.067073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.992492+0000","last_change":"2024-03-26T05:21:30.832264+0000","last_active":"2024-03-26T05:22:09.992492+0000","last_peered":"2024-03-26T05:22:09.992492+0000","last_clean":"2024-03-26T05:22:09.992492+0000","last_became_active":"2024-03-26T05:21:30.831939+0000","last_became_peered":"2024-03-26T05:21:30.831939+0000","last_unstale":"2024-03-26T05:22:09.992492+0000","last_undegraded":"2024-03-26T05:22:09.992492+0000","last_fullsized":"2024-03-26T05:22:09.992492+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:15:29.806546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'20","reported_seq":62,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809012+0000","last_change":"2024-03-26T05:21:32.843704+0000","last_active":"2024-03-26T05:22:09.809012+0000","last_peered":"2024-03-26T05:22:09.809012+0000","last_clean":"2024-03-26T05:22:09.809012+0000","last_became_active":"2024-03-26T05:21:32.843350+0000","last_became_peered":"2024-03-26T05:21:32.843350+0000","last_unstale":"2024-03-26T05:22:09.809012+0000","last_undegraded":"2024-03-26T05:22:09.809012+0000","last_fullsized":"2024-03-26T05:22:09.809012+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:07:38.742244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889435+0000","last_change":"2024-03-26T05:21:34.845930+0000","last_active":"2024-03-26T05:22:45.889435+0000","last_peered":"2024-03-26T05:22:45.889435+0000","last_clean":"2024-03-26T05:22:45.889435+0000","last_became_active":"2024-03-26T05:21:34.845606+0000","last_became_peered":"2024-03-26T05:21:34.845606+0000","last_unstale":"2024-03-26T05:22:45.889435+0000","last_undegraded":"2024-03-26T05:22:45.889435+0000","last_fullsized":"2024-03-26T05:22:45.889435+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:12:25.747607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808826+0000","last_change":"2024-03-26T05:21:36.856977+0000","last_active":"2024-03-26T05:22:09.808826+0000","last_peered":"2024-03-26T05:22:09.808826+0000","last_clean":"2024-03-26T05:22:09.808826+0000","last_became_active":"2024-03-26T05:21:36.856677+0000","last_became_peered":"2024-03-26T05:21:36.856677+0000","last_unstale":"2024-03-26T05:22:09.808826+0000","last_undegraded":"2024-03-26T05:22:09.808826+0000","last_fullsized":"2024-03-26T05:22:09.808826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:48:36.894140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809833+0000","last_change":"2024-03-26T05:21:30.826982+0000","last_active":"2024-03-26T05:22:09.809833+0000","last_peered":"2024-03-26T05:22:09.809833+0000","last_clean":"2024-03-26T05:22:09.809833+0000","last_became_active":"2024-03-26T05:21:30.826076+0000","last_became_peered":"2024-03-26T05:21:30.826076+0000","last_unstale":"2024-03-26T05:22:09.809833+0000","last_undegraded":"2024-03-26T05:22:09.809833+0000","last_fullsized":"2024-03-26T05:22:09.809833+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:23:19.558990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'30","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810173+0000","last_change":"2024-03-26T05:21:33.316202+0000","last_active":"2024-03-26T05:22:09.810173+0000","last_peered":"2024-03-26T05:22:09.810173+0000","last_clean":"2024-03-26T05:22:09.810173+0000","last_became_active":"2024-03-26T05:21:33.315916+0000","last_became_peered":"2024-03-26T05:21:33.315916+0000","last_unstale":"2024-03-26T05:22:09.810173+0000","last_undegraded":"2024-03-26T05:22:09.810173+0000","last_fullsized":"2024-03-26T05:22:09.810173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:45.078692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808736+0000","last_change":"2024-03-26T05:21:34.843885+0000","last_active":"2024-03-26T05:22:09.808736+0000","last_peered":"2024-03-26T05:22:09.808736+0000","last_clean":"2024-03-26T05:22:09.808736+0000","last_became_active":"2024-03-26T05:21:34.843557+0000","last_became_peered":"2024-03-26T05:21:34.843557+0000","last_unstale":"2024-03-26T05:22:09.808736+0000","last_undegraded":"2024-03-26T05:22:09.808736+0000","last_fullsized":"2024-03-26T05:22:09.808736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:57:50.589496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.911826+0000","last_change":"2024-03-26T05:21:36.854664+0000","last_active":"2024-03-26T05:22:09.911826+0000","last_peered":"2024-03-26T05:22:09.911826+0000","last_clean":"2024-03-26T05:22:09.911826+0000","last_became_active":"2024-03-26T05:21:36.854402+0000","last_became_peered":"2024-03-26T05:21:36.854402+0000","last_unstale":"2024-03-26T05:22:09.911826+0000","last_undegraded":"2024-03-26T05:22:09.911826+0000","last_fullsized":"2024-03-26T05:22:09.911826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:26:51.369897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889821+0000","last_change":"2024-03-26T05:21:34.856402+0000","last_active":"2024-03-26T05:22:45.889821+0000","last_peered":"2024-03-26T05:22:45.889821+0000","last_clean":"2024-03-26T05:22:45.889821+0000","last_became_active":"2024-03-26T05:21:34.856106+0000","last_became_peered":"2024-03-26T05:21:34.856106+0000","last_unstale":"2024-03-26T05:22:45.889821+0000","last_undegraded":"2024-03-26T05:22:45.889821+0000","last_fullsized":"2024-03-26T05:22:45.889821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:17:05.578941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810087+0000","last_change":"2024-03-26T05:21:30.826413+0000","last_active":"2024-03-26T05:22:09.810087+0000","last_peered":"2024-03-26T05:22:09.810087+0000","last_clean":"2024-03-26T05:22:09.810087+0000","last_became_active":"2024-03-26T05:21:30.825733+0000","last_became_peered":"2024-03-26T05:21:30.825733+0000","last_unstale":"2024-03-26T05:22:09.810087+0000","last_undegraded":"2024-03-26T05:22:09.810087+0000","last_fullsized":"2024-03-26T05:22:09.810087+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:39:19.042121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991633+0000","last_change":"2024-03-26T05:21:32.853211+0000","last_active":"2024-03-26T05:22:09.991633+0000","last_peered":"2024-03-26T05:22:09.991633+0000","last_clean":"2024-03-26T05:22:09.991633+0000","last_became_active":"2024-03-26T05:21:32.852901+0000","last_became_peered":"2024-03-26T05:21:32.852901+0000","last_unstale":"2024-03-26T05:22:09.991633+0000","last_undegraded":"2024-03-26T05:22:09.991633+0000","last_fullsized":"2024-03-26T05:22:09.991633+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:06:01.791707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806763+0000","last_change":"2024-03-26T05:21:37.148681+0000","last_active":"2024-03-26T05:22:09.806763+0000","last_peered":"2024-03-26T05:22:09.806763+0000","last_clean":"2024-03-26T05:22:09.806763+0000","last_became_active":"2024-03-26T05:21:37.148395+0000","last_became_peered":"2024-03-26T05:21:37.148395+0000","last_unstale":"2024-03-26T05:22:09.806763+0000","last_undegraded":"2024-03-26T05:22:09.806763+0000","last_fullsized":"2024-03-26T05:22:09.806763+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:28:35.544419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.912601+0000","last_change":"2024-03-26T05:21:34.846788+0000","last_active":"2024-03-26T05:22:09.912601+0000","last_peered":"2024-03-26T05:22:09.912601+0000","last_clean":"2024-03-26T05:22:09.912601+0000","last_became_active":"2024-03-26T05:21:34.846427+0000","last_became_peered":"2024-03-26T05:21:34.846427+0000","last_unstale":"2024-03-26T05:22:09.912601+0000","last_undegraded":"2024-03-26T05:22:09.912601+0000","last_fullsized":"2024-03-26T05:22:09.912601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:17:36.698608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.808850+0000","last_change":"2024-03-26T05:21:30.831683+0000","last_active":"2024-03-26T05:22:09.808850+0000","last_peered":"2024-03-26T05:22:09.808850+0000","last_clean":"2024-03-26T05:22:09.808850+0000","last_became_active":"2024-03-26T05:21:30.830180+0000","last_became_peered":"2024-03-26T05:21:30.830180+0000","last_unstale":"2024-03-26T05:22:09.808850+0000","last_undegraded":"2024-03-26T05:22:09.808850+0000","last_fullsized":"2024-03-26T05:22:09.808850+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:02:15.235458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'17","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809603+0000","last_change":"2024-03-26T05:21:32.849241+0000","last_active":"2024-03-26T05:22:09.809603+0000","last_peered":"2024-03-26T05:22:09.809603+0000","last_clean":"2024-03-26T05:22:09.809603+0000","last_became_active":"2024-03-26T05:21:32.848484+0000","last_became_peered":"2024-03-26T05:21:32.848484+0000","last_unstale":"2024-03-26T05:22:09.809603+0000","last_undegraded":"2024-03-26T05:22:09.809603+0000","last_fullsized":"2024-03-26T05:22:09.809603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:49:47.299249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.806644+0000","last_change":"2024-03-26T05:21:36.856534+0000","last_active":"2024-03-26T05:22:09.806644+0000","last_peered":"2024-03-26T05:22:09.806644+0000","last_clean":"2024-03-26T05:22:09.806644+0000","last_became_active":"2024-03-26T05:21:36.856230+0000","last_became_peered":"2024-03-26T05:21:36.856230+0000","last_unstale":"2024-03-26T05:22:09.806644+0000","last_undegraded":"2024-03-26T05:22:09.806644+0000","last_fullsized":"2024-03-26T05:22:09.806644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:24:52.728213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.991784+0000","last_change":"2024-03-26T05:21:34.852423+0000","last_active":"2024-03-26T05:22:09.991784+0000","last_peered":"2024-03-26T05:22:09.991784+0000","last_clean":"2024-03-26T05:22:09.991784+0000","last_became_active":"2024-03-26T05:21:34.852117+0000","last_became_peered":"2024-03-26T05:21:34.852117+0000","last_unstale":"2024-03-26T05:22:09.991784+0000","last_undegraded":"2024-03-26T05:22:09.991784+0000","last_fullsized":"2024-03-26T05:22:09.991784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:34.664073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.990677+0000","last_change":"2024-03-26T05:21:30.831931+0000","last_active":"2024-03-26T05:22:09.990677+0000","last_peered":"2024-03-26T05:22:09.990677+0000","last_clean":"2024-03-26T05:22:09.990677+0000","last_became_active":"2024-03-26T05:21:30.831627+0000","last_became_peered":"2024-03-26T05:21:30.831627+0000","last_unstale":"2024-03-26T05:22:09.990677+0000","last_undegraded":"2024-03-26T05:22:09.990677+0000","last_fullsized":"2024-03-26T05:22:09.990677+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:09:16.212042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810397+0000","last_change":"2024-03-26T05:21:32.849542+0000","last_active":"2024-03-26T05:22:09.810397+0000","last_peered":"2024-03-26T05:22:09.810397+0000","last_clean":"2024-03-26T05:22:09.810397+0000","last_became_active":"2024-03-26T05:21:32.848581+0000","last_became_peered":"2024-03-26T05:21:32.848581+0000","last_unstale":"2024-03-26T05:22:09.810397+0000","last_undegraded":"2024-03-26T05:22:09.810397+0000","last_fullsized":"2024-03-26T05:22:09.810397+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:39:35.696152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809947+0000","last_change":"2024-03-26T05:21:36.856981+0000","last_active":"2024-03-26T05:22:09.809947+0000","last_peered":"2024-03-26T05:22:09.809947+0000","last_clean":"2024-03-26T05:22:09.809947+0000","last_became_active":"2024-03-26T05:21:36.856638+0000","last_became_peered":"2024-03-26T05:21:36.856638+0000","last_unstale":"2024-03-26T05:22:09.809947+0000","last_undegraded":"2024-03-26T05:22:09.809947+0000","last_fullsized":"2024-03-26T05:22:09.809947+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:35.834367+0000","last_clean_scrub_stamp":"2024-03-26T05:21:35.834367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:32:25.901388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:45.889437+0000","last_change":"2024-03-26T05:21:34.848338+0000","last_active":"2024-03-26T05:22:45.889437+0000","last_peered":"2024-03-26T05:22:45.889437+0000","last_clean":"2024-03-26T05:22:45.889437+0000","last_became_active":"2024-03-26T05:21:34.848028+0000","last_became_peered":"2024-03-26T05:21:34.848028+0000","last_unstale":"2024-03-26T05:22:45.889437+0000","last_undegraded":"2024-03-26T05:22:45.889437+0000","last_fullsized":"2024-03-26T05:22:45.889437+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:33.830725+0000","last_clean_scrub_stamp":"2024-03-26T05:21:33.830725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:46:31.513830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.810463+0000","last_change":"2024-03-26T05:21:30.831172+0000","last_active":"2024-03-26T05:22:09.810463+0000","last_peered":"2024-03-26T05:22:09.810463+0000","last_clean":"2024-03-26T05:22:09.810463+0000","last_became_active":"2024-03-26T05:21:30.830830+0000","last_became_peered":"2024-03-26T05:21:30.830830+0000","last_unstale":"2024-03-26T05:22:09.810463+0000","last_undegraded":"2024-03-26T05:22:09.810463+0000","last_fullsized":"2024-03-26T05:22:09.810463+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:29.802551+0000","last_clean_scrub_stamp":"2024-03-26T05:21:29.802551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:28:16.455447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-26T05:22:09.809675+0000","last_change":"2024-03-26T05:21:32.843994+0000","last_active":"2024-03-26T05:22:09.809675+0000","last_peered":"2024-03-26T05:22:09.809675+0000","last_clean":"2024-03-26T05:22:09.809675+0000","last_became_active":"2024-03-26T05:21:32.843629+0000","last_became_peered":"2024-03-26T05:21:32.843629+0000","last_unstale":"2024-03-26T05:22:09.809675+0000","last_undegraded":"2024-03-26T05:22:09.809675+0000","last_fullsized":"2024-03-26T05:22:09.809675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T05:21:31.813088+0000","last_clean_scrub_stamp":"2024-03-26T05:21:31.813088+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:31:56.321038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":589,"ondisk_log_size":589,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32456,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712696,"statfs":{"total":95995035648,"available":95961800704,"internally_reserved":0,"allocated":1310720,"data_stored":872343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1298432,"data_stored":872179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":753664,"data_stored":282776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1314816,"data_stored":872085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":700416,"data_stored":281258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713244,"statfs":{"total":95995035648,"available":95962361856,"internally_reserved":0,"allocated":745472,"data_stored":283100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27732,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717420,"statfs":{"total":95995035648,"available":95966638080,"internally_reserved":0,"allocated":729088,"data_stored":281926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31876,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713276,"statfs":{"total":95995035648,"available":95962394624,"internally_reserved":0,"allocated":712704,"data_stored":282247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T05:23:01.193 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T05:23:01.193 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T05:23:01.194 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T05:23:01.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T05:23:01.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:00 smithi164 ceph-mon[33554]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:01.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:00 smithi164 ceph-mon[33554]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T05:23:01.436 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T05:23:01.621 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 systemd[1]: Starting Ceph grafana.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:23:01.892 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 podman[63592]: 2024-03-26 05:23:01.533497225 +0000 UTC m=+0.017056536 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-26T05:23:01.892 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 podman[63592]: 2024-03-26 05:23:01.633612513 +0000 UTC m=+0.117171857 container create 5523b342ffffe806c0b7025a818c7829d18913838787e336ee4249b91dbfadd2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a, io.openshift.expose-services=, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=ubi8-container, version=8.5, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, distribution-scope=public, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git) 2024-03-26T05:23:01.892 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 podman[63592]: 2024-03-26 05:23:01.810532022 +0000 UTC m=+0.294091324 container init 5523b342ffffe806c0b7025a818c7829d18913838787e336ee4249b91dbfadd2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, release=239.1651231664, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-26T05:23:01.892 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 podman[63592]: 2024-03-26 05:23:01.813090657 +0000 UTC m=+0.296649971 container start 5523b342ffffe806c0b7025a818c7829d18913838787e336ee4249b91dbfadd2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a, vendor=Red Hat, Inc., distribution-scope=public, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, io.openshift.expose-services=, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664) 2024-03-26T05:23:02.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:23:02.144 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 bash[63592]: 5523b342ffffe806c0b7025a818c7829d18913838787e336ee4249b91dbfadd2 2024-03-26T05:23:02.144 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 systemd[1]: Started Ceph grafana.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971358506Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.97153891Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971557692Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971564964Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971570909Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.97157703Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971582982Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971588569Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971594229Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971599684Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T05:23:02.145 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971605501Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T05:23:02.146 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971610877Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T05:23:02.146 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.971617037Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T05:23:02.146 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=settings t=2024-03-26T05:23:01.97162282Z level=info msg="App mode production" 2024-03-26T05:23:02.146 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=sqlstore t=2024-03-26T05:23:01.971685972Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T05:23:02.146 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=sqlstore t=2024-03-26T05:23:01.971702627Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T05:23:02.146 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.972555948Z level=info msg="Starting DB migrations" 2024-03-26T05:23:02.146 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.972630985Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T05:23:02.147 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.973169737Z level=info msg="Executing migration" id="create user table" 2024-03-26T05:23:02.147 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.973613679Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T05:23:02.147 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.974458437Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T05:23:02.147 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.975406551Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.976229946Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.977053719Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.977954495Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.978809288Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.979630903Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.980492839Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.981382741Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.982205682Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.982779226Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.983605486Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.984465111Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.985316884Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.98574518Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T05:23:02.148 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.986638962Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.987509661Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.988698774Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.989177727Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.99001053Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.99094894Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.991376853Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.994236769Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.99685982Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:01.99969225Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.002392135Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.005184626Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.007902741Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T05:23:02.149 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.010503942Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.01321217Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.016040621Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.018695493Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.0213551Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.024416174Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.026991638Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.027407218Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.030247666Z level=info msg="Executing migration" id="create star table" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.030894357Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.033101981Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.035806569Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.03824872Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T05:23:02.150 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.040886097Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.043575948Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.044063257Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.046828298Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.049461165Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.051875359Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.052235754Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.054938271Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.057723207Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.060496004Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.061090698Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.063899482Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.066653555Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T05:23:02.151 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.069226745Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.069681832Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.072419289Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.073370113Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.074065923Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.074503595Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.075279431Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.075848372Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.076386284Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.077170173Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.077634769Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.078163221Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.078962372Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T05:23:02.152 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.079922638Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.080610469Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.080937968Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.081842001Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.082632624Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.083553207Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.084288109Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.085120885Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.085997232Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.086915685Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.087293057Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.088117573Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.088544463Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T05:23:02.153 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.089479995Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.09031244Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.09078738Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.09172917Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.092471138Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.093279824Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.093873563Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.094392822Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.095202287Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.095570327Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.096081842Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.096993099Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.097493359Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T05:23:02.154 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.100270937Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.103011181Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.105557652Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.106026543Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.10900302Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.111443121Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.114188963Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.114735949Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.117713281Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.120455829Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.123254983Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.123595189Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.126505807Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.129200332Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T05:23:02.155 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.131945876Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.132311371Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.133162908Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.133533111Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.134141738Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.137001358Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.137434301Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.140389135Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.14327583Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.145917048Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.146344902Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.149231429Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.149681151Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T05:23:02.156 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.152217116Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T05:23:02.157 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.155018318Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T05:23:02.157 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.157880878Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T05:23:02.395 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.161046866Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.163865062Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.164203606Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.166739365Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.167048664Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.169639379Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.17020441Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.172531218Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.175299203Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.177936339Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.180210046Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T05:23:02.396 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.182935341Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.185392226Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.187895126Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.190503879Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.193376461Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.19599853Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.198472868Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.201007118Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.203620179Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.206483828Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.209044034Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.211686572Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.214418551Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T05:23:02.397 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.214858147Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.217530777Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.21828195Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.219137215Z level=info msg="Executing migration" id="create session table" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.21962672Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.220429688Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.220744937Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.221142677Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.221982328Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.222908438Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.223200776Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.223476934Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T05:23:02.398 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.223753084Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T05:23:02.399 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.224164099Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T05:23:02.400 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.22442787Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T05:23:02.400 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.225031584Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T05:23:02.400 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.225854467Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T05:23:02.400 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.226472414Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T05:23:02.400 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.227365544Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T05:23:02.400 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.227678514Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T05:23:02.400 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.228175158Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.229127186Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.230061852Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.230817423Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.231622042Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.232143576Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.232574437Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.23300818Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.233428877Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.234833726Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.235244748Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.235686753Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.236028792Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.236402251Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T05:23:02.401 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.236756604Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.237421874Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.238045995Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.238686531Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.2393585Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.239806825Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.240120759Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.240389177Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.240807607Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.241270691Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.241620097Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.24203221Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.242442486Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T05:23:02.402 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.243107555Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.243785813Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.244139328Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.244587499Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.245026355Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.245697576Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.246044368Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.246460661Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.246908306Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.24725061Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.247696234Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.248500334Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T05:23:02.403 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.248976177Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.249443184Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.24995378Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.250828355Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.251120781Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.251756088Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.252186542Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.252824914Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.25323243Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.253734508Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T05:23:02.404 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.254637423Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T05:23:02.405 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.256190707Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T05:23:02.405 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.257069616Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T05:23:02.405 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.258087382Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.258803484Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.259790642Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.260531185Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.261342094Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.262345698Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.263145586Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.264058032Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.264991154Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.265784308Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.266838342Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.267712308Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.270571087Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T05:23:02.406 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.273433154Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.276210237Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.279066421Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.281887863Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.284659437Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.287549034Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.29033494Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.293021606Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.293537638Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.296456766Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.299302726Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.302177966Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.30516763Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.308008204Z level=info msg="Executing migration" id="create team table" 2024-03-26T05:23:02.407 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.31096012Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.313842943Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.316637081Z level=info msg="Executing migration" id="create team member table" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.317159402Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.320093372Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.322684943Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.325543162Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.32826756Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.331031663Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.333664353Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.336354531Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.339053085Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.341735687Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.344179102Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T05:23:02.408 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.346873177Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.349720425Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.352531606Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.355112048Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.355531638Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.358239471Z level=info msg="Executing migration" id="create tag table" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.358655581Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.360895878Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.361555994Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.362443257Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.363151913Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.365067234Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.365983165Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.366529071Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T05:23:02.409 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.36729012Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.367674118Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.368131341Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.369101123Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.369620343Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.370427356Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.371308573Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.37206615Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.372827164Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.373631609Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.374505543Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.375371037Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.37617284Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.377021074Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T05:23:02.410 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.37798079Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.378909774Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.379418952Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.380260607Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.381108981Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.38209813Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.38259674Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.383598984Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.384312244Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.385150837Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.385703793Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.386175166Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.387075736Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.388084173Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T05:23:02.411 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.39089085Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.391348904Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.394211178Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.394707399Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.39711538Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.399839809Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.400156983Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.403003241Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.403475135Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.405885312Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.408693169Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T05:23:02.412 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.409136995Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.411902764Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.412347847Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.415026235Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.417855876Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.418280912Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.420964628Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.424456309Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.428001373Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.430769875Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T05:23:02.667 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.431247672Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T05:23:02.668 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.434180717Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T05:23:02.668 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.436711133Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T05:23:02.668 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.43916644Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T05:23:02.668 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.439769124Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T05:23:02.668 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.442564991Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T05:23:02.668 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.445026316Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T05:23:02.668 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.447867894Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.448712424Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.451565683Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.454162859Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.45684397Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.457689148Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.460432642Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.461140796Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.462000499Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.462826394Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T05:23:02.669 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.463820609Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.464498104Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.46543415Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.466030157Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.466804227Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.467760571Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.468608997Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.469473206Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.470357942Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.471226666Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.472166793Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.472576155Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.473450883Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T05:23:02.670 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.474277039Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.475152723Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.475590756Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.476441308Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.477297489Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.478257447Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.479254974Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.480066889Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.481006717Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.481734704Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.482187953Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.483031269Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.483445348Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.483862807Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T05:23:02.671 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.483939937Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.484217113Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.484808242Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.485430946Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.485546515Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.485667463Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.486077896Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.486948832Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.49001656Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.492708035Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.493253689Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.495954054Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.499061866Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.501881912Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.502436849Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T05:23:02.672 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.50519331Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.509570542Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.512454272Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.512799369Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.517026922Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.521213279Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.524311287Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.524762413Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.527651173Z level=info msg="Executing migration" id="create permission table" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.530192493Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.532950485Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.533409949Z level=info msg="Executing migration" id="create role table" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.536079414Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.538837795Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.541567209Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T05:23:02.673 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.542184039Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.543065142Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.543931932Z level=info msg="Executing migration" id="create team role table" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.544965753Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.545507201Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.546485726Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.54719586Z level=info msg="Executing migration" id="create user role table" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.548146697Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.548738877Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.54961122Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.550571062Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.551478348Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T05:23:02.674 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.552121202Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.552978691Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.55394904Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.554453697Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.554934674Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.555365903Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.555784684Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.556185161Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.556641472Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.557606572Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.55804591Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.558489314Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.558777491Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.559201458Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.55963962Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T05:23:02.675 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.560045419Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T05:23:02.676 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.560364093Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.560712481Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.561124902Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.56156551Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.56254485Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.562861774Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.563288848Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.563720453Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.56415973Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.565152341Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.565598743Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.566032656Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T05:23:02.677 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.566126737Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.566440215Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.566528701Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.566796309Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.567196821Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.56757924Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.568054698Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.568521181Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.568943479Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.569369487Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.569731399Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.570160307Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.570562303Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.571022461Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.57144783Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T05:23:02.678 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.574506979Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.575587805Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.576596294Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.576936346Z level=info msg="Executing migration" id="add share column" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.577919651Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.578343495Z level=info msg="Executing migration" id="create file table" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.578818389Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.57931744Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.579815459Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.580653306Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.58168995Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.582033948Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.582902578Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T05:23:02.679 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.583228031Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.583722733Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.584754915Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.585097243Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.585551142Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.586442033Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.587352378Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.58823861Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.589328052Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.59040778Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.59126106Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.600967551Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.601420342Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.604395143Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.60799354Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.610803332Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T05:23:02.680 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.61115892Z level=info msg="Executing migration" id="create folder table" 2024-03-26T05:23:02.681 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.61393943Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T05:23:02.681 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.614408653Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T05:23:02.681 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.616848845Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T05:23:02.681 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.6197471Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T05:23:02.681 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=migrator t=2024-03-26T05:23:02.620245598Z level=info msg="migrations completed" performed=485 skipped=0 duration=647.63686ms 2024-03-26T05:23:02.681 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=sqlstore t=2024-03-26T05:23:02.620619265Z level=info msg="Created default organization" 2024-03-26T05:23:02.681 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=plugin.loader t=2024-03-26T05:23:02.677524788Z level=info msg="Plugin registered" pluginID=input 2024-03-26T05:23:02.949 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=plugin.loader t=2024-03-26T05:23:02.705566747Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T05:23:02.949 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=plugin.loader t=2024-03-26T05:23:02.705589267Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T05:23:02.949 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=secrets t=2024-03-26T05:23:02.705624306Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T05:23:02.949 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=query_data t=2024-03-26T05:23:02.707717193Z level=info msg="Query Service initialization" 2024-03-26T05:23:02.949 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=live.push_http t=2024-03-26T05:23:02.710059612Z level=info msg="Live Push Gateway initialization" 2024-03-26T05:23:03.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-mon[33554]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:03.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:03.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:02 smithi164 ceph-mon[33554]: Deploying daemon node-exporter.a on smithi136 2024-03-26T05:23:03.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[38106]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:03.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:03.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.225 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[38106]: Deploying daemon node-exporter.a on smithi136 2024-03-26T05:23:03.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[29560]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:03.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:03.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:03.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:02 smithi136 ceph-mon[29560]: Deploying daemon node-exporter.a on smithi136 2024-03-26T05:23:03.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 -- 172.21.15.136:0/1647020365 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5201073c0 msgr2=0x7ff52010ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 --2- 172.21.15.136:0/1647020365 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5201073c0 0x7ff52010ef90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b0a0 tx=0x7ff51800b3b0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 -- 172.21.15.136:0/1647020365 shutdown_connections 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 --2- 172.21.15.136:0/1647020365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52006aa00 0x7ff52006ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 --2- 172.21.15.136:0/1647020365 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5201073c0 0x7ff52010ef90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 --2- 172.21.15.136:0/1647020365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff52006b3b0 0x7ff520106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 -- 172.21.15.136:0/1647020365 >> 172.21.15.136:0/1647020365 conn(0x7ff520077e40 msgr2=0x7ff520078250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 -- 172.21.15.136:0/1647020365 shutdown_connections 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.479+0000 7ff524da6700 1 -- 172.21.15.136:0/1647020365 wait complete. 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff524da6700 1 Processor -- start 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff524da6700 1 -- start start 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff524da6700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52006aa00 0x7ff520197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff524da6700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52006b3b0 0x7ff5201982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:23:03.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff524da6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5201073c0 0x7ff52019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff524da6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52019cc90 con 0x7ff5201073c0 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff524da6700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52019ce00 con 0x7ff52006aa00 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff524da6700 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52019cfc0 con 0x7ff52006b3b0 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff51dd9b700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52006b3b0 0x7ff5201982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff51ed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5201073c0 0x7ff52019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff51e59c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52006aa00 0x7ff520197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff51dd9b700 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52006b3b0 0x7ff5201982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:52362/0 (socket says 172.21.15.136:52362) 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.480+0000 7ff51dd9b700 1 -- 172.21.15.136:0/528168632 learned_addr learned my addr 172.21.15.136:0/528168632 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff51ed9d700 1 -- 172.21.15.136:0/528168632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52006b3b0 msgr2=0x7ff5201982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff51ed9d700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52006b3b0 0x7ff5201982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff51ed9d700 1 -- 172.21.15.136:0/528168632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52006aa00 msgr2=0x7ff520197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff51ed9d700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52006aa00 0x7ff520197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff51ed9d700 1 -- 172.21.15.136:0/528168632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518009d00 con 0x7ff5201073c0 2024-03-26T05:23:03.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff51dd9b700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52006b3b0 0x7ff5201982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:23:03.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff51e59c700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52006aa00 0x7ff520197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:23:03.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff51ed9d700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5201073c0 0x7ff52019c680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b070 tx=0x7ff51800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:23:03.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff50f7fe700 1 -- 172.21.15.136:0/528168632 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51800e040 con 0x7ff5201073c0 2024-03-26T05:23:03.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff50f7fe700 1 -- 172.21.15.136:0/528168632 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ff518004e60 con 0x7ff5201073c0 2024-03-26T05:23:03.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52019d240 con 0x7ff5201073c0 2024-03-26T05:23:03.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff50f7fe700 1 -- 172.21.15.136:0/528168632 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51801dd90 con 0x7ff5201073c0 2024-03-26T05:23:03.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.481+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff520101430 con 0x7ff5201073c0 2024-03-26T05:23:03.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.482+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5201037c0 con 0x7ff5201073c0 2024-03-26T05:23:03.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.483+0000 7ff50f7fe700 1 -- 172.21.15.136:0/528168632 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7ff51801d500 con 0x7ff5201073c0 2024-03-26T05:23:03.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.483+0000 7ff50f7fe700 1 --2- 172.21.15.136:0/528168632 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff50806da80 0x7ff50806ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:23:03.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.483+0000 7ff50f7fe700 1 -- 172.21.15.136:0/528168632 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6259+0+0 (secure 0 0 0) 0x7ff51808e800 con 0x7ff5201073c0 2024-03-26T05:23:03.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.483+0000 7ff51e59c700 1 --2- 172.21.15.136:0/528168632 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff50806da80 0x7ff50806ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:23:03.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.484+0000 7ff51e59c700 1 --2- 172.21.15.136:0/528168632 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff50806da80 0x7ff50806ff40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff510009730 tx=0x7ff510012040 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:23:03.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.488+0000 7ff50f7fe700 1 -- 172.21.15.136:0/528168632 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff51805bca0 con 0x7ff5201073c0 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=infra.usagestats.collector t=2024-03-26T05:23:03.326868499Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=provisioning.datasources t=2024-03-26T05:23:03.327888215Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=provisioning.datasources t=2024-03-26T05:23:03.338867645Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=provisioning.alerting t=2024-03-26T05:23:03.349430369Z level=info msg="starting to provision alerting" 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=provisioning.alerting t=2024-03-26T05:23:03.349448608Z level=info msg="finished to provision alerting" 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=ngalert.state.manager t=2024-03-26T05:23:03.349520306Z level=info msg="Warming state cache for startup" 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=grafanaStorageLogger t=2024-03-26T05:23:03.349610357Z level=info msg="storage starting" 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=ngalert.state.manager t=2024-03-26T05:23:03.349668512Z level=info msg="State cache has been initialized" states=0 duration=147.052µs 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=ticker t=2024-03-26T05:23:03.349793583Z level=info msg=starting first_tick=2024-03-26T05:23:10Z 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=ngalert.multiorg.alertmanager t=2024-03-26T05:23:03.349825303Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=http.server t=2024-03-26T05:23:03.351332795Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T05:23:03.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=sqlstore.transactions t=2024-03-26T05:23:03.360212752Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T05:23:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.744+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff520068d40 con 0x7ff5201073c0 2024-03-26T05:23:03.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.744+0000 7ff50f7fe700 1 -- 172.21.15.136:0/528168632 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff51805b830 con 0x7ff5201073c0 2024-03-26T05:23:03.735 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-26T05:23:03.735 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T05:23:03.735 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:03 smithi136 bash[72955]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T05:23:03.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff50806da80 msgr2=0x7ff50806ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:03.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 --2- 172.21.15.136:0/528168632 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff50806da80 0x7ff50806ff40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff510009730 tx=0x7ff510012040 comp rx=0 tx=0).stop 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5201073c0 msgr2=0x7ff52019c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5201073c0 0x7ff52019c680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b070 tx=0x7ff51800bdd0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 shutdown_connections 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 --2- 172.21.15.136:0/528168632 >> v2:172.21.15.136:6800/3380174081 conn(0x7ff50806da80 0x7ff50806ff40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff52006aa00 0x7ff520197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5201073c0 0x7ff52019c680 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 --2- 172.21.15.136:0/528168632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff52006b3b0 0x7ff5201982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 >> 172.21.15.136:0/528168632 conn(0x7ff520077e40 msgr2=0x7ff5200ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 shutdown_connections 2024-03-26T05:23:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-26T05:23:03.747+0000 7ff524da6700 1 -- 172.21.15.136:0/528168632 wait complete. 2024-03-26T05:23:04.061 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/528168632' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T05:23:04.062 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/528168632' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T05:23:04.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/528168632' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T05:23:04.323 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:04 smithi136 bash[72955]: Getting image source signatures 2024-03-26T05:23:04.323 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:04 smithi136 bash[72955]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T05:23:04.324 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:04 smithi136 bash[72955]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T05:23:04.324 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:04 smithi136 bash[72955]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T05:23:04.931 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T05:23:04.932 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T05:23:04.932 INFO:teuthology.run_tasks:Running task workunit... 2024-03-26T05:23:04.949 INFO:tasks.workunit:Pulling workunits from ref 9f04c91b77e4619963d41730d8cf756e3a6152de 2024-03-26T05:23:04.950 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-26T05:23:04.950 DEBUG:teuthology.orchestra.run.smithi136:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-26T05:23:04.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:23:04.967 INFO:teuthology.orchestra.run.smithi136.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-26T05:23:04.967 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T05:23:05.021 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-26T05:23:05.021 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-26T05:23:05.076 INFO:tasks.workunit:timeout=3h 2024-03-26T05:23:05.076 INFO:tasks.workunit:cleanup=True 2024-03-26T05:23:05.077 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9f04c91b77e4619963d41730d8cf756e3a6152de 2024-03-26T05:23:05.132 INFO:tasks.workunit.client.0.smithi136.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-26T05:23:05.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:04 smithi164 ceph-mon[33554]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:05.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:04 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:05.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:04 smithi136 ceph-mon[38106]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:05.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:04 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:05.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:04 smithi136 ceph-mon[29560]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:05.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:04 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:06.323 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 bash[72955]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T05:23:06.324 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 bash[72955]: Writing manifest to image destination 2024-03-26T05:23:06.824 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 podman[72955]: 2024-03-26 05:23:06.51781942 +0000 UTC m=+2.958982439 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T05:23:06.824 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 podman[72955]: 2024-03-26 05:23:06.607882089 +0000 UTC m=+3.049045097 container create 938b3b4f121bd496fe5280f3808c73eba60ec68fcaf87fb7546bda963e5e654d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:06.824 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 podman[72955]: 2024-03-26 05:23:06.818222698 +0000 UTC m=+3.259385707 container init 938b3b4f121bd496fe5280f3808c73eba60ec68fcaf87fb7546bda963e5e654d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:06.824 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 podman[72955]: 2024-03-26 05:23:06.820778688 +0000 UTC m=+3.261941695 container start 938b3b4f121bd496fe5280f3808c73eba60ec68fcaf87fb7546bda963e5e654d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.824Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.824Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.825Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.825Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.825Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.825Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.825Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.825Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.825Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.825Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T05:23:06.825 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T05:23:06.826 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T05:23:06.827 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T05:23:06.828 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T05:23:06.829 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a[73089]: ts=2024-03-26T05:23:06.826Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T05:23:07.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:06 smithi164 ceph-mon[33554]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:07.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-mon[38106]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:07.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:06 smithi136 ceph-mon[29560]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:07.228 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 bash[72955]: 938b3b4f121bd496fe5280f3808c73eba60ec68fcaf87fb7546bda963e5e654d 2024-03-26T05:23:07.228 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:23:06 smithi136 systemd[1]: Started Ceph node-exporter.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:23:07.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:23:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:08.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-mon[38106]: Deploying daemon node-exporter.b on smithi164 2024-03-26T05:23:08.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:07 smithi136 ceph-mon[29560]: Deploying daemon node-exporter.b on smithi164 2024-03-26T05:23:08.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:07 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:07 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:07 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:08.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:07 smithi164 ceph-mon[33554]: Deploying daemon node-exporter.b on smithi164 2024-03-26T05:23:08.468 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:08 smithi164 bash[63871]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T05:23:08.967 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:08 smithi164 bash[63871]: Getting image source signatures 2024-03-26T05:23:08.968 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:08 smithi164 bash[63871]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T05:23:08.968 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:08 smithi164 bash[63871]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T05:23:08.968 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:08 smithi164 bash[63871]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T05:23:09.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:09 smithi136 ceph-mon[29560]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:09.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:09 smithi136 ceph-mon[38106]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:09.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:09 smithi164 ceph-mon[33554]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:10.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:10.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:10.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:10.468 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:09 smithi164 bash[63871]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T05:23:10.468 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 bash[63871]: Writing manifest to image destination 2024-03-26T05:23:10.808 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 podman[63871]: 2024-03-26 05:23:10.434816086 +0000 UTC m=+2.344514334 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T05:23:10.808 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 podman[63871]: 2024-03-26 05:23:10.533493768 +0000 UTC m=+2.443192009 container create 9864a66ae61cb2d9a9e4e01614b6d576fc7214e90efc1e179057dd6a2d7e736a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:23:10.808 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 podman[63871]: 2024-03-26 05:23:10.735275095 +0000 UTC m=+2.644973334 container init 9864a66ae61cb2d9a9e4e01614b6d576fc7214e90efc1e179057dd6a2d7e736a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 podman[63871]: 2024-03-26 05:23:10.737985856 +0000 UTC m=+2.647684096 container start 9864a66ae61cb2d9a9e4e01614b6d576fc7214e90efc1e179057dd6a2d7e736a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.741Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.741Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T05:23:10.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T05:23:10.810 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T05:23:10.811 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.742Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.743Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.743Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T05:23:10.812 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.743Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T05:23:10.813 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.743Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T05:23:10.813 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.743Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T05:23:10.813 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b[63920]: ts=2024-03-26T05:23:10.743Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T05:23:10.813 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 bash[63871]: 9864a66ae61cb2d9a9e4e01614b6d576fc7214e90efc1e179057dd6a2d7e736a 2024-03-26T05:23:11.217 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:23:10 smithi164 systemd[1]: Started Ceph node-exporter.b for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:23:11.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:11 smithi164 ceph-mon[33554]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:11.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:11 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:11.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[38106]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[29560]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:11 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:12.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:23:11 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:23:13.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:13 smithi136 ceph-mon[38106]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:13.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:13.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:13 smithi136 ceph-mon[29560]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:13.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:13.416 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:13 smithi164 ceph-mon[33554]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:13.416 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:15.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:15 smithi136 ceph-mon[38106]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:15.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:15.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:15.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:15.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:15 smithi136 ceph-mon[29560]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:15.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:15.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:15.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:15.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:15 smithi164 ceph-mon[33554]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:16.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:16.513 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.513 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:16.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:16.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:16 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:16 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:16 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:16 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:16 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:16.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:16 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:17.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:17 smithi136 ceph-mon[29560]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:17.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:17 smithi136 ceph-mon[29560]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T05:23:17.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:17 smithi136 ceph-mon[29560]: Reconfiguring daemon alertmanager.a on smithi136 2024-03-26T05:23:17.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:23:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:17 smithi136 ceph-mon[38106]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:17 smithi136 ceph-mon[38106]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T05:23:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:17 smithi136 ceph-mon[38106]: Reconfiguring daemon alertmanager.a on smithi136 2024-03-26T05:23:17.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:17 smithi164 ceph-mon[33554]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:17 smithi164 ceph-mon[33554]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T05:23:17.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:17 smithi164 ceph-mon[33554]: Reconfiguring daemon alertmanager.a on smithi136 2024-03-26T05:23:19.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:19 smithi164 ceph-mon[33554]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:19.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:19 smithi136 ceph-mon[29560]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:19.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:19 smithi136 ceph-mon[38106]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:19.574 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:19 smithi136 systemd[1]: Stopping Ceph alertmanager.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:23:20.042 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:19 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[71925]: ts=2024-03-26T05:23:19.768Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:23:20.042 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:19 smithi136 podman[73813]: 2024-03-26 05:23:19.788469684 +0000 UTC m=+0.129011083 container died 8c131f9b360d36befa0994e7d5ee9cded3f0ab53e7536c6ef1817d2353f5f529 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:20.574 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:20 smithi136 podman[73813]: 2024-03-26 05:23:20.310111374 +0000 UTC m=+0.650652815 container cleanup 8c131f9b360d36befa0994e7d5ee9cded3f0ab53e7536c6ef1817d2353f5f529 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:20.574 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:20 smithi136 bash[73813]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T05:23:20.574 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:20 smithi136 podman[73827]: 2024-03-26 05:23:20.511429143 +0000 UTC m=+0.719350922 container remove 8c131f9b360d36befa0994e7d5ee9cded3f0ab53e7536c6ef1817d2353f5f529 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:20.869 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:20 smithi136 podman[73827]: 2024-03-26 05:23:20.625551562 +0000 UTC m=+0.833473297 volume remove bbc4fe6cb98099e8002a16daf278fca977551c9b513e6352d2cbf605f477b21b 2024-03-26T05:23:21.186 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:20 smithi136 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T05:23:21.186 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:20 smithi136 systemd[1]: Stopped Ceph alertmanager.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:23:21.186 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:20 smithi136 systemd[1]: Starting Ceph alertmanager.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:23:21.186 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 podman[73921]: 2024-03-26 05:23:21.096129638 +0000 UTC m=+0.115431976 volume create af34d3e06d85a7df01c6884440d08df4fa9fe90e6367e4c84f79397e6d959e58 2024-03-26T05:23:21.186 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 podman[73921]: 2024-03-26 05:23:20.998907736 +0000 UTC m=+0.018210035 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T05:23:21.186 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 podman[73921]: 2024-03-26 05:23:21.196449593 +0000 UTC m=+0.215751882 container create 021b2158ebf0ba82fd61d18a9a36aed170e0bf50888f86c8388020ca7f4b73c2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:21.514 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-mon[38106]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:21.514 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-mon[29560]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:21.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:21 smithi164 ceph-mon[33554]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:21.823 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 podman[73921]: 2024-03-26 05:23:21.524416143 +0000 UTC m=+0.543718436 container init 021b2158ebf0ba82fd61d18a9a36aed170e0bf50888f86c8388020ca7f4b73c2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 podman[73921]: 2024-03-26 05:23:21.527232997 +0000 UTC m=+0.546535287 container start 021b2158ebf0ba82fd61d18a9a36aed170e0bf50888f86c8388020ca7f4b73c2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:21.539Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:21.539Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:21.540Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.136 port=9094 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:21.541Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:21.564Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:21.565Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:21.567Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:21.567Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 bash[73921]: 021b2158ebf0ba82fd61d18a9a36aed170e0bf50888f86c8388020ca7f4b73c2 2024-03-26T05:23:21.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:21 smithi136 systemd[1]: Started Ceph alertmanager.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:23:22.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:23:21 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:23:22.768 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:22 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:22.768 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:22 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:22.768 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:22 smithi164 ceph-mon[33554]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T05:23:23.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:22 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:23.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:22 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:23.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:22 smithi136 ceph-mon[38106]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T05:23:23.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:22 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:23.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:22 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:23.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:22 smithi136 ceph-mon[29560]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T05:23:23.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:23 smithi136 ceph-mon[38106]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:23.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:23.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:23 smithi136 ceph-mon[38106]: Reconfiguring daemon prometheus.a on smithi164 2024-03-26T05:23:23.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:23 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:23.541Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000046932s 2024-03-26T05:23:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:23 smithi136 ceph-mon[29560]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:23.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:23 smithi136 ceph-mon[29560]: Reconfiguring daemon prometheus.a on smithi164 2024-03-26T05:23:23.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:23 smithi164 ceph-mon[33554]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:23.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:23.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:23 smithi164 ceph-mon[33554]: Reconfiguring daemon prometheus.a on smithi164 2024-03-26T05:23:25.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:25 smithi164 ceph-mon[33554]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:25.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:25 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:26.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:25 smithi136 ceph-mon[38106]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:26.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:25 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:26.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:25 smithi136 ceph-mon[29560]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:26.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:25 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:27.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:23:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:27.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:27 smithi164 ceph-mon[33554]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:27 smithi136 ceph-mon[38106]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:27 smithi136 ceph-mon[29560]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:29.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:29 smithi164 ceph-mon[33554]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:29.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:29 smithi164 systemd[1]: Stopping Ceph prometheus.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:23:30.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:29 smithi136 ceph-mon[38106]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:29 smithi136 ceph-mon[29560]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:31.218 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:23:31.218 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.916Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.917Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T05:23:31.219 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.917Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T05:23:31.220 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[61665]: ts=2024-03-26T05:23:30.917Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T05:23:31.220 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:30 smithi164 podman[64823]: 2024-03-26 05:23:30.927240253 +0000 UTC m=+1.211205028 container died 593a7e65c964633e7b2e112e0404e8d55ef4a0c82bfa53f9dd07e5dffce5c1ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:31.823 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:31 smithi136 ceph-mon[38106]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:31.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:31 smithi136 ceph-mon[29560]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:31.824 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:23:31 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:23:31.543Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00224887s 2024-03-26T05:23:31.910 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:31 smithi164 ceph-mon[33554]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:31.910 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:31 smithi164 podman[64823]: 2024-03-26 05:23:31.691425397 +0000 UTC m=+1.975390160 container cleanup 593a7e65c964633e7b2e112e0404e8d55ef4a0c82bfa53f9dd07e5dffce5c1ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:31.910 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:31 smithi164 bash[64823]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T05:23:32.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:23:31 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:23:32.218 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:31 smithi164 podman[64837]: 2024-03-26 05:23:31.967725343 +0000 UTC m=+1.041356276 container remove 593a7e65c964633e7b2e112e0404e8d55ef4a0c82bfa53f9dd07e5dffce5c1ed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:32.218 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T05:23:32.218 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 systemd[1]: Stopped Ceph prometheus.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:23:32.218 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.115s CPU time. 2024-03-26T05:23:32.218 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 systemd[1]: Starting Ceph prometheus.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:23:32.570 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 podman[64931]: 2024-03-26 05:23:32.223684966 +0000 UTC m=+0.018866799 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T05:23:32.571 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 podman[64931]: 2024-03-26 05:23:32.343640629 +0000 UTC m=+0.138822505 container create 0d842328995192eff452599d748502b48d2f076decb7a0e2e69a8eed68b31dd4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:32.571 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 podman[64931]: 2024-03-26 05:23:32.554102131 +0000 UTC m=+0.349283964 container init 0d842328995192eff452599d748502b48d2f076decb7a0e2e69a8eed68b31dd4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:32.571 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 podman[64931]: 2024-03-26 05:23:32.556786744 +0000 UTC m=+0.351968584 container start 0d842328995192eff452599d748502b48d2f076decb7a0e2e69a8eed68b31dd4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:23:32.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.581Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T05:23:32.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.581Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T05:23:32.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.581Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi164 (none))" 2024-03-26T05:23:32.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.581Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T05:23:32.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.581Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T05:23:32.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.583Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T05:23:32.968 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.583Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.584Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.585Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.587Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.587Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.197µs 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.587Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.587Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.587Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.587Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.48µs wal_replay_duration=427.292µs wbl_replay_duration=110ns total_replay_duration=463.505µs 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.588Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.588Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.588Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.606Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.228271ms db_storage=974ns remote_storage=1.118µs web_handler=345ns query_engine=428ns scrape=383.92µs scrape_sd=71.275µs notify=15.688µs notify_sd=11.128µs rules=17.552111ms tracing=5.964µs 2024-03-26T05:23:32.969 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.606Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T05:23:32.970 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:23:32.606Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T05:23:32.970 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 bash[64931]: 0d842328995192eff452599d748502b48d2f076decb7a0e2e69a8eed68b31dd4 2024-03-26T05:23:32.970 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:23:32 smithi164 systemd[1]: Started Ceph prometheus.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STOPPING 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STOPPED 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STARTING 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Serving on http://:::9283 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STARTED 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STOPPING 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STOPPED 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STARTING 2024-03-26T05:23:33.028 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Serving on http://:::9283 2024-03-26T05:23:33.029 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STARTED 2024-03-26T05:23:33.029 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STOPPING 2024-03-26T05:23:33.029 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:23:33.029 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STOPPED 2024-03-26T05:23:33.029 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:32 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:32] ENGINE Bus STARTING 2024-03-26T05:23:33.029 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:33] ENGINE Serving on http://:::9283 2024-03-26T05:23:33.029 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: [26/Mar/2024:05:23:33] ENGINE Bus STARTED 2024-03-26T05:23:33.807 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:33.807 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:33.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:23:33.809 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:23:33.809 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:23:33.809 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:23:33.809 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:33.809 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:23:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:23:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:23:35.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:35 smithi136 ceph-mon[38106]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:35.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:35 smithi136 ceph-mon[29560]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:35.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:35 smithi164 ceph-mon[33554]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:37.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:37 smithi136 ceph-mon[38106]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:37.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:37 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:37.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:37 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:37 smithi136 ceph-mon[29560]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:37 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:37.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:37 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:37.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:37 smithi164 ceph-mon[33554]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:37.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:37 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:37.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:37 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:38 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:38 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:38 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:38.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:38 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:38.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:38 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:23:38.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:38 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:23:39.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:39 smithi164 ceph-mon[33554]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:39.474 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:39 smithi136 ceph-mon[29560]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:39.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:39 smithi136 ceph-mon[38106]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:40.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:40 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:40.490 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:40 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:40.491 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:40 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:41.467 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:41 smithi164 ceph-mon[33554]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:41.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:41 smithi136 ceph-mon[38106]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:41.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:41 smithi136 ceph-mon[29560]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:42.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:23:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:23:43.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:43 smithi164 ceph-mon[33554]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:43.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:43.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:43 smithi136 ceph-mon[38106]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:43.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:43.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:43 smithi136 ceph-mon[29560]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:43.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:45 smithi164 ceph-mon[33554]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:45.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:45 smithi136 ceph-mon[38106]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:45.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:45 smithi136 ceph-mon[29560]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:47.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:47 smithi164 ceph-mon[33554]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:47.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:47 smithi136 ceph-mon[38106]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:47.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:47 smithi136 ceph-mon[29560]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:23:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:49.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:49 smithi164 ceph-mon[33554]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:49.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:49 smithi136 ceph-mon[38106]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:49.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:49 smithi136 ceph-mon[29560]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:51.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:51 smithi164 ceph-mon[33554]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:51.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:51 smithi136 ceph-mon[38106]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:51.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:51 smithi136 ceph-mon[29560]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:52.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:23:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:23:53.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:53 smithi164 ceph-mon[33554]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:53.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:53.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:53 smithi136 ceph-mon[38106]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:53.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:53 smithi136 ceph-mon[29560]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:53.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:23:55.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:55 smithi164 ceph-mon[33554]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:55.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:55 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:55.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:55 smithi136 ceph-mon[38106]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:55.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:55 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:55.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:55 smithi136 ceph-mon[29560]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:55.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:55 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:23:57.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:57 smithi164 ceph-mon[33554]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:57.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:57 smithi136 ceph-mon[38106]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:57.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:23:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:23:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:23:57.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:57 smithi136 ceph-mon[29560]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:23:59.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:23:59 smithi164 ceph-mon[33554]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:59.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:23:59 smithi136 ceph-mon[38106]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:23:59.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:23:59 smithi136 ceph-mon[29560]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:01.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:01 smithi164 ceph-mon[33554]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:01.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:01 smithi136 ceph-mon[38106]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:01.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:01 smithi136 ceph-mon[29560]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:02.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:24:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:24:03.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:03 smithi164 ceph-mon[33554]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:03.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:03 smithi136 ceph-mon[38106]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:03.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:03 smithi136 ceph-mon[29560]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:03.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:05.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:05 smithi164 ceph-mon[33554]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:05.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:05 smithi136 ceph-mon[38106]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:05.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:05 smithi136 ceph-mon[29560]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:07.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:07 smithi164 ceph-mon[33554]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:07.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:07 smithi136 ceph-mon[38106]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:07.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:24:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:24:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:07.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:07 smithi136 ceph-mon[29560]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:09 smithi164 ceph-mon[33554]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:09.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:09 smithi136 ceph-mon[38106]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:09.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:09 smithi136 ceph-mon[29560]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:10.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:10.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:10.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:10.644 INFO:tasks.workunit.client.0.smithi136.stderr:Note: switching to '9f04c91b77e4619963d41730d8cf756e3a6152de'. 2024-03-26T05:24:10.644 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-03-26T05:24:10.644 INFO:tasks.workunit.client.0.smithi136.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-26T05:24:10.644 INFO:tasks.workunit.client.0.smithi136.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-26T05:24:10.644 INFO:tasks.workunit.client.0.smithi136.stderr:state without impacting any branches by switching back to a branch. 2024-03-26T05:24:10.644 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-03-26T05:24:10.644 INFO:tasks.workunit.client.0.smithi136.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr: git switch -c 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr:Or undo this operation with: 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr: git switch - 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-03-26T05:24:10.645 INFO:tasks.workunit.client.0.smithi136.stderr:HEAD is now at 9f04c91b77e Merge PR #56391 into squid 2024-03-26T05:24:10.648 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-26T05:24:10.679 INFO:tasks.workunit.client.0.smithi136.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-26T05:24:10.681 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T05:24:10.681 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-26T05:24:11.441 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-26T05:24:11.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:11 smithi164 ceph-mon[33554]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:11.508 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-26T05:24:11.548 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T05:24:11.549 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T05:24:11.550 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-26T05:24:11.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:11 smithi136 ceph-mon[38106]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:11.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:11 smithi136 ceph-mon[29560]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:24:11.588 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T05:24:11.591 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-26T05:24:11.591 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-26T05:24:11.647 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-26T05:24:11.648 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-26T05:24:11.649 DEBUG:teuthology.orchestra.run.smithi136:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-26T05:24:11.714 INFO:tasks.workunit.client.0.smithi136.stderr:+ parallel=1 2024-03-26T05:24:11.714 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' '' = --serial ']' 2024-03-26T05:24:11.714 INFO:tasks.workunit.client.0.smithi136.stderr:+ crimson=0 2024-03-26T05:24:11.714 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' '' = --crimson ']' 2024-03-26T05:24:11.714 INFO:tasks.workunit.client.0.smithi136.stderr:+ color= 2024-03-26T05:24:11.714 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -t 1 ']' 2024-03-26T05:24:11.715 INFO:tasks.workunit.client.0.smithi136.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-26T05:24:11.715 INFO:tasks.workunit.client.0.smithi136.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T05:24:11.715 INFO:tasks.workunit.client.0.smithi136.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T05:24:11.716 INFO:tasks.workunit.client.0.smithi136.stderr:+ declare -A pids 2024-03-26T05:24:11.716 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.716 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.716 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_aio 2024-03-26T05:24:11.716 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_aio' 2024-03-26T05:24:11.716 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_aio 2024-03-26T05:24:11.716 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.717 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_aio 2024-03-26T05:24:11.717 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74693 2024-03-26T05:24:11.717 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_aio on pid 74693' 2024-03-26T05:24:11.717 INFO:tasks.workunit.client.0.smithi136.stdout:test api_aio on pid 74693 2024-03-26T05:24:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74693 2024-03-26T05:24:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_aio_pp 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_aio_pp' 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_aio_pp 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.721 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.722 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.723 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_aio_pp 2024-03-26T05:24:11.723 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74700 2024-03-26T05:24:11.723 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_aio_pp on pid 74700' 2024-03-26T05:24:11.723 INFO:tasks.workunit.client.0.smithi136.stdout:test api_aio_pp on pid 74700 2024-03-26T05:24:11.726 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74700 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_io 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_io' 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_io 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.727 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_io 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74709 2024-03-26T05:24:11.730 INFO:tasks.workunit.client.0.smithi136.stdout:test api_io on pid 74709 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_io on pid 74709' 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74709 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_io_pp 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_io_pp' 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_io_pp 2024-03-26T05:24:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.736 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.737 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_io_pp 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74720 2024-03-26T05:24:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_io_pp on pid 74720' 2024-03-26T05:24:11.739 INFO:tasks.workunit.client.0.smithi136.stdout:test api_io_pp on pid 74720 2024-03-26T05:24:11.742 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74720 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_asio 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_asio' 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_asio 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.743 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.744 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_asio 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74738 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_asio on pid 74738' 2024-03-26T05:24:11.749 INFO:tasks.workunit.client.0.smithi136.stdout:test api_asio on pid 74738 2024-03-26T05:24:11.751 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74738 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_list 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_list' 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_list 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.752 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.753 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.756 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.759 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74693/exe 2024-03-26T05:24:11.759 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_list 2024-03-26T05:24:11.759 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74752 2024-03-26T05:24:11.759 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_list on pid 74752' 2024-03-26T05:24:11.759 INFO:tasks.workunit.client.0.smithi136.stdout:test api_list on pid 74752 2024-03-26T05:24:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74752 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_lock 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_lock' 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.764 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_lock 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.765 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.766 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.767 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_aio: /' 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.769 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.770 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.771 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74700/exe 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_lock 2024-03-26T05:24:11.772 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74772 2024-03-26T05:24:11.773 INFO:tasks.workunit.client.0.smithi136.stdout:test api_lock on pid 74772 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_lock on pid 74772' 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74772 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_lock_pp 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_lock_pp' 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.776 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.777 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.778 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.782 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_lock_pp 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.783 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_lock_pp 2024-03-26T05:24:11.784 INFO:tasks.workunit.client.0.smithi136.stdout:test api_lock_pp on pid 74792 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74792 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_lock_pp on pid 74792' 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74792 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_misc 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_misc' 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_misc 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.790 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.791 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.792 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74709/exe 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.795 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.797 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_misc 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74813 2024-03-26T05:24:11.798 INFO:tasks.workunit.client.0.smithi136.stdout:test api_misc on pid 74813 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_misc on pid 74813' 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74813 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.804 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_io: /' 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74720/exe 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_io.log 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_misc_pp 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_misc_pp' 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.806 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.807 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.808 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_misc_pp 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.810 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.811 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.812 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_misc_pp 2024-03-26T05:24:11.812 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.812 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.812 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.812 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74831 2024-03-26T05:24:11.812 INFO:tasks.workunit.client.0.smithi136.stdout:test api_misc_pp on pid 74831 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_misc_pp on pid 74831' 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74831 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.815 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_tier_pp 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_tier_pp' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74738/exe 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74752/exe 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_tier_pp 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.819 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.820 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-26T05:24:11.822 INFO:tasks.workunit.client.0.smithi136.stdout:test api_tier_pp on pid 74855 2024-03-26T05:24:11.825 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.825 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.825 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_tier_pp 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74855 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_tier_pp on pid 74855' 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74855 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.826 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_list.log 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_asio: /' 2024-03-26T05:24:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_list: /' 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_pool 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.830 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_pool' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.832 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.834 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_pool 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.835 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74792/exe 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.836 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.837 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.837 INFO:tasks.workunit.client.0.smithi136.stdout:test api_pool on pid 74876 2024-03-26T05:24:11.838 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.838 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.838 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.838 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.838 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.838 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_pool 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74876 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_pool on pid 74876' 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74876 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_snapshots 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_snapshots' 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.839 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_snapshots 2024-03-26T05:24:11.840 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-26T05:24:11.840 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.840 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.840 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.840 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.840 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.840 INFO:tasks.workunit.client.0.smithi136.stdout:test api_snapshots on pid 74884 2024-03-26T05:24:11.843 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_snapshots 2024-03-26T05:24:11.843 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.843 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74884 2024-03-26T05:24:11.843 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_snapshots on pid 74884' 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74884 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_snapshots_pp 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.844 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_snapshots_pp' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_snapshots_pp 2024-03-26T05:24:11.845 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.847 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.849 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.849 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.849 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.849 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.849 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.849 INFO:tasks.workunit.client.0.smithi136.stdout:test api_snapshots_pp on pid 74905 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_snapshots_pp 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74905 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_snapshots_pp on pid 74905' 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74905 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.852 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.853 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74772/exe 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.854 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.855 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_lock: /' 2024-03-26T05:24:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74813/exe 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_stat 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_stat' 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_stat 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_misc: /' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.861 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_stat 2024-03-26T05:24:11.862 INFO:tasks.workunit.client.0.smithi136.stdout:test api_stat on pid 74957 2024-03-26T05:24:11.865 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74957 2024-03-26T05:24:11.865 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_stat on pid 74957' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=74957 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.866 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.867 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.869 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_stat_pp 2024-03-26T05:24:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_stat_pp' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.871 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.873 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_stat_pp 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.874 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.875 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.876 INFO:tasks.workunit.client.0.smithi136.stdout:test api_stat_pp on pid 75018 2024-03-26T05:24:11.880 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_stat_pp 2024-03-26T05:24:11.880 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75018 2024-03-26T05:24:11.880 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_stat_pp on pid 75018' 2024-03-26T05:24:11.880 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75018 2024-03-26T05:24:11.880 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.880 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.880 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_watch_notify 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.881 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_watch_notify' 2024-03-26T05:24:11.882 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.884 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.885 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_watch_notify 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.886 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.887 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.888 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_watch_notify 2024-03-26T05:24:11.889 INFO:tasks.workunit.client.0.smithi136.stdout:test api_watch_notify on pid 75052 2024-03-26T05:24:11.893 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.893 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75052 2024-03-26T05:24:11.893 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_watch_notify on pid 75052' 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75052 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74884/exe 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.895 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.898 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_watch_notify_pp 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_watch_notify_pp' 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.899 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.900 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74855/exe 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.901 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.902 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.903 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.904 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.905 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.905 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74831/exe 2024-03-26T05:24:11.905 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_watch_notify_pp 2024-03-26T05:24:11.905 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-26T05:24:11.905 INFO:tasks.workunit.client.0.smithi136.stdout:test api_watch_notify_pp on pid 75076 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_watch_notify_pp 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.911 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75076 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_watch_notify_pp on pid 75076' 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75076 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.913 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_cmd 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_cmd' 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74905/exe 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74876/exe 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.914 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.915 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.916 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.917 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_cmd 2024-03-26T05:24:11.918 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.919 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_cmd 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75111 2024-03-26T05:24:11.920 INFO:tasks.workunit.client.0.smithi136.stdout:test api_cmd on pid 75111 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_cmd on pid 75111' 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75111 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_pool: /' 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.923 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_cmd_pp 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-26T05:24:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_cmd_pp' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_cmd_pp 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.926 INFO:tasks.workunit.client.0.smithi136.stdout:test api_cmd_pp on pid 75136 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_cmd_pp 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75136 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_cmd_pp on pid 75136' 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75136 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.929 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.933 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_service 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_service' 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_service 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.934 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.935 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_service 2024-03-26T05:24:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:test api_service on pid 75156 2024-03-26T05:24:11.941 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75156 2024-03-26T05:24:11.941 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_service on pid 75156' 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75156 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_service_pp 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_service_pp' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.943 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.944 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.945 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.946 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75018/exe 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.947 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_service_pp 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/74957/exe 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.951 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.951 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.951 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.951 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.951 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.951 INFO:tasks.workunit.client.0.smithi136.stdout:test api_service_pp on pid 75195 2024-03-26T05:24:11.952 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.952 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.952 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.952 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.952 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_service_pp 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75195 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_service_pp on pid 75195' 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75195 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_stat: /' 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_c_write_operations 2024-03-26T05:24:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.954 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_c_write_operations' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75052/exe 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.955 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.956 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.956 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.956 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-26T05:24:11.956 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-26T05:24:11.956 INFO:tasks.workunit.client.0.smithi136.stdout:test api_c_write_operations on pid 75211 2024-03-26T05:24:11.961 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.961 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_c_write_operations 2024-03-26T05:24:11.961 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.961 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.961 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_c_write_operations 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75211 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_c_write_operations on pid 75211' 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.962 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75211 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_c_read_operations 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.963 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_c_read_operations' 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.964 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.965 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_c_read_operations 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-26T05:24:11.966 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.968 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.972 INFO:tasks.workunit.client.0.smithi136.stdout:test api_c_read_operations on pid 75274 2024-03-26T05:24:11.977 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_c_read_operations 2024-03-26T05:24:11.977 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75274 2024-03-26T05:24:11.977 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_c_read_operations on pid 75274' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75274 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:11.978 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.979 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.980 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75156/exe 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.981 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.982 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.983 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s api_cls_remote_reads 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.984 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_service: /' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' api_cls_remote_reads' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_service.log 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.986 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo api_cls_remote_reads 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.987 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.988 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=api_cls_remote_reads 2024-03-26T05:24:11.990 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75111/exe 2024-03-26T05:24:11.990 INFO:tasks.workunit.client.0.smithi136.stdout:test api_cls_remote_reads on pid 75328 2024-03-26T05:24:11.994 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75328 2024-03-26T05:24:11.994 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test api_cls_remote_reads on pid 75328' 2024-03-26T05:24:11.994 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75328 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:11.995 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_cmd: /' 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:11.996 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:11.997 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s list_parallel 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' list_parallel' 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.003 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75076/exe 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.004 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo list_parallel 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.007 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.008 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=list_parallel 2024-03-26T05:24:12.008 INFO:tasks.workunit.client.0.smithi136.stdout:test list_parallel on pid 75375 2024-03-26T05:24:12.015 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75375 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test list_parallel on pid 75375' 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75375 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s open_pools_parallel 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.016 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' open_pools_parallel' 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.017 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.018 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.019 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75211/exe 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.020 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-26T05:24:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75274/exe 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.023 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.024 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo open_pools_parallel 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.025 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.026 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.027 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75136/exe 2024-03-26T05:24:12.027 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=open_pools_parallel 2024-03-26T05:24:12.027 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75435 2024-03-26T05:24:12.027 INFO:tasks.workunit.client.0.smithi136.stdout:test open_pools_parallel on pid 75435 2024-03-26T05:24:12.030 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test open_pools_parallel on pid 75435' 2024-03-26T05:24:12.030 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75435 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.031 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.032 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.033 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75195/exe 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s delete_pools_parallel 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' delete_pools_parallel' 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo delete_pools_parallel 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.036 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.037 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.038 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.038 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.038 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.038 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.038 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.038 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.038 INFO:tasks.workunit.client.0.smithi136.stdout:test delete_pools_parallel on pid 75465 2024-03-26T05:24:12.043 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=delete_pools_parallel 2024-03-26T05:24:12.043 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.043 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.043 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75465 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test delete_pools_parallel on pid 75465' 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75465 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.044 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s cls 2024-03-26T05:24:12.045 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' cls' 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo cls 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-26T05:24:12.046 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75328/exe 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.047 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.049 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.050 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=cls 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75500 2024-03-26T05:24:12.051 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test cls on pid 75500' 2024-03-26T05:24:12.052 INFO:tasks.workunit.client.0.smithi136.stdout:test cls on pid 75500 2024-03-26T05:24:12.052 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75500 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.053 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s cmd 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' cmd' 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo cmd 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.054 INFO:tasks.workunit.client.0.smithi136.stdout:test cmd on pid 75514 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=cmd 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75514 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test cmd on pid 75514' 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75514 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.059 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.060 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.061 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75375/exe 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s handler_error 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' handler_error' 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.064 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-26T05:24:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo handler_error 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ list_parallel: /' 2024-03-26T05:24:12.067 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.068 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.069 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.069 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.069 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.069 INFO:tasks.workunit.client.0.smithi136.stdout:test handler_error on pid 75560 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=handler_error 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75560 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test handler_error on pid 75560' 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75560 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.073 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s io 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' io' 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-26T05:24:12.074 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.075 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.076 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.077 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo io 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.078 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.079 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.080 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.080 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.080 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.080 INFO:tasks.workunit.client.0.smithi136.stdout:test io on pid 75616 2024-03-26T05:24:12.083 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=io 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75616 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test io on pid 75616' 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75616 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.085 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.086 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.087 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75435/exe 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-26T05:24:12.088 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s ec_io 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' ec_io' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo ec_io 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.091 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.092 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.093 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.094 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-26T05:24:12.095 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=ec_io 2024-03-26T05:24:12.096 INFO:tasks.workunit.client.0.smithi136.stdout:test ec_io on pid 75651 2024-03-26T05:24:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75651 2024-03-26T05:24:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test ec_io on pid 75651' 2024-03-26T05:24:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75651 2024-03-26T05:24:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75465/exe 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s list 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' list' 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.100 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo list 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.101 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.102 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.103 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.104 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75500/exe 2024-03-26T05:24:12.105 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ cls: /' 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-26T05:24:12.108 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_cls 2024-03-26T05:24:12.109 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.109 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.109 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.109 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.109 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.109 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.109 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.109 INFO:tasks.workunit.client.0.smithi136.stdout:test list on pid 75696 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=list 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75696 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test list on pid 75696' 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75696 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_cls.log 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75560/exe 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.114 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.115 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.116 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.117 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.118 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.119 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s ec_list 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' ec_list' 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_handler_error 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo ec_list 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ handler_error: /' 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.122 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.124 INFO:tasks.workunit.client.0.smithi136.stdout:test ec_list on pid 75747 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=ec_list 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75747 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test ec_list on pid 75747' 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75747 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s misc 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' misc' 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.127 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.127 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.127 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.127 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.127 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo misc 2024-03-26T05:24:12.127 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75616/exe 2024-03-26T05:24:12.127 INFO:tasks.workunit.client.0.smithi136.stdout:test misc on pid 75754 2024-03-26T05:24:12.131 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=misc 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75754 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test misc on pid 75754' 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75754 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s pool 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' pool' 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75514/exe 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.132 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.133 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ io: /' 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_io 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_cmd 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.138 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.138 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo pool 2024-03-26T05:24:12.138 INFO:tasks.workunit.client.0.smithi136.stdout:test pool on pid 75778 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=pool 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75778 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test pool on pid 75778' 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75778 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s read_operations 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_io.log 2024-03-26T05:24:12.143 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' read_operations' 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.144 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.145 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.146 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo read_operations 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ cmd: /' 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.149 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.150 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=read_operations 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75827 2024-03-26T05:24:12.151 INFO:tasks.workunit.client.0.smithi136.stdout:test read_operations on pid 75827 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test read_operations on pid 75827' 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75827 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s snapshots 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.157 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.158 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' snapshots' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.159 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.160 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo snapshots 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.161 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.162 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.163 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.163 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.163 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.163 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.163 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.163 INFO:tasks.workunit.client.0.smithi136.stdout:test snapshots on pid 75875 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=snapshots 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75875 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test snapshots on pid 75875' 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75875 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s watch_notify 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.165 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' watch_notify' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75754/exe 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.166 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.167 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.168 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.169 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.170 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.171 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo watch_notify 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.172 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.173 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.173 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.173 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.173 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ misc: /' 2024-03-26T05:24:12.173 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=watch_notify 2024-03-26T05:24:12.173 INFO:tasks.workunit.client.0.smithi136.stdout:test watch_notify on pid 75904 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75904 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test watch_notify on pid 75904' 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75904 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75651/exe 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:++ printf %25s write_operations 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.175 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+ r=' write_operations' 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_misc.log 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:++ awk '{print $1}' 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:++ echo write_operations 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.177 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.177 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.177 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.177 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-26T05:24:12.177 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_misc 2024-03-26T05:24:12.177 INFO:tasks.workunit.client.0.smithi136.stderr:+ ff=write_operations 2024-03-26T05:24:12.177 INFO:tasks.workunit.client.0.smithi136.stdout:test write_operations on pid 75932 2024-03-26T05:24:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75932 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'test write_operations on pid 75932' 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+ pids[$f]=75932 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+ ret=0 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74855 2024-03-26T05:24:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74855 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_ec_io 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ ec_io: /' 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:+ . /etc/bashrc 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -z '' ']' 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++ BASHRCSOURCED=Y 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++ shopt -q login_shell 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:+++ umask 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ SHELL=/bin/bash 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75696/exe 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T05:24:12.199 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T05:24:12.200 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.201 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.202 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.204 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.205 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_list 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_list.log 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.206 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ list: /' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.207 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75778/exe 2024-03-26T05:24:12.208 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75747/exe 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_pool.log 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_pool 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.212 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.213 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.214 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_ec_list 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ pool: /' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:+++ return 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T05:24:12.215 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export S_COLORS=auto 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ S_COLORS=auto 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ ec_list: /' 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.218 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75827/exe 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.221 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75932/exe 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ read_operations: /' 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_read_operations 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.224 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75875/exe 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_write_operations 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ write_operations: /' 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_snapshots 2024-03-26T05:24:12.225 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ snapshots: /' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/locale 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:++++ LANG=en_US.UTF-8 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T05:24:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:+++ LANG=en_US.UTF-8 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:+++ unset LANG_backup config 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n '' ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' dumb = linux ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/less.sh 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' '' ']' 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T05:24:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+++++ readlink /proc/75904/exe 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:++++ basename /usr/bin/bash 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_declare='declare -f' 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+++ which_opt=-f 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export which_declare 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+++ export -f which 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset i 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:++ unset -f pathmunge 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+ export PATH 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+ unset rc 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph_test_neorados_watch_notify 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-26T05:24:12.229 INFO:tasks.workunit.client.0.smithi136.stderr:+ sed 's/^/ watch_notify: /' 2024-03-26T05:24:12.230 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:24:11 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:24:12.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:12.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:12.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/970748271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3119355251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/21326660' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T05:24:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:13 smithi164 ceph-mon[33554]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/970748271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3119355251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/21326660' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[38106]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:14.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/970748271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3119355251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/21326660' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T05:24:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:13 smithi136 ceph-mon[29560]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T05:24:17.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:24:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:24:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3003057331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1027303946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1420108388' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:19.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3003057331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1027303946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1420108388' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3003057331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1027303946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1420108388' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:20.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:20.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/560934285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/274651958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2857831464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/143476809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1298358617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2457825477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1713077913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4083411389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/292595320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1375283821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2660893759' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/560934285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/274651958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2857831464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/143476809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1298358617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2457825477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1713077913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4083411389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/292595320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1375283821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T05:24:20.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2660893759' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:20.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/560934285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/274651958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2857831464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/143476809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1298358617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2457825477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1713077913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4083411389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/292595320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1375283821' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2660893759' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: pgmap v67: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: pgmap v68: 356 pgs: 169 creating+peering, 134 active+clean, 53 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: pgmap v69: 356 pgs: 169 creating+peering, 134 active+clean, 53 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: pgmap v72: 1092 pgs: 169 creating+peering, 134 active+clean, 789 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2107963243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1349030646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3953232739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:21.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:21.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1960741666' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: pgmap v67: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: pgmap v68: 356 pgs: 169 creating+peering, 134 active+clean, 53 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: pgmap v69: 356 pgs: 169 creating+peering, 134 active+clean, 53 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:21.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: pgmap v72: 1092 pgs: 169 creating+peering, 134 active+clean, 789 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2107963243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1349030646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3953232739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T05:24:21.580 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:21.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1960741666' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:21.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [----------] Global test environment set-up. 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (0 ms) 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (5 ms) 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (4 ms) 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-26T05:24:21.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3 ms) 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-03-26T05:24:21.633 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [----------] 12 tests from AsioRados (23 ms total) 2024-03-26T05:24:21.634 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: 2024-03-26T05:24:21.634 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [----------] Global test environment tear-down 2024-03-26T05:24:21.634 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (9823 ms total) 2024-03-26T05:24:21.634 INFO:tasks.workunit.client.0.smithi136.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-26T05:24:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: pgmap v67: 356 pgs: 224 unknown, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: pgmap v68: 356 pgs: 169 creating+peering, 134 active+clean, 53 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: pgmap v69: 356 pgs: 169 creating+peering, 134 active+clean, 53 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi136-75990-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi136-75924-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi136-74761-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: pgmap v72: 1092 pgs: 169 creating+peering, 134 active+clean, 789 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2107963243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75440-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi136-74781-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi136-74810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi136-74847-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi136-74915-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi136-74824-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1349030646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3953232739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:21.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:21.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:21.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1960741666' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [----------] Global test environment set-up. 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosList.ListObjects (1330 ms) 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (53 ms) 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo1 2024-03-26T05:24:21.939 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo2 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo3 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo4 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo5 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo1 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: myset foo6,foo7 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo7 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo6 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns1:foo4 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns1:foo5 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns2:foo7 2024-03-26T05:24:21.940 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns2:foo6 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns1:foo1 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: :foo1 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: :foo2 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: :foo3 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (94 ms) 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 1 0 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 10 0 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 13 0 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 7 0 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 14 0 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 0 0 2024-03-26T05:24:21.941 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 15 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 11 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 5 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 8 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 6 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 3 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 4 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 12 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 9 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 2 0 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (62 ms) 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: x cursor=MIN 2024-03-26T05:24:21.942 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=1 cursor=12:02547ec2:::1:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=10 cursor=12:52ea6a34:::10:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=13 cursor=12:566253c9:::13:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=7 cursor=12:5c6b0b28:::7:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=14 cursor=12:62a1935d:::14:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=0 cursor=12:6cac518f:::0:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=15 cursor=12:863748b0:::15:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=11 cursor=12:89d3ae78:::11:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=5 cursor=12:b29083e3:::5:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=8 cursor=12:bd63b0f1:::8:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=6 cursor=12:c4fdafeb:::6:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=3 cursor=12:cfc208b3:::3:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=4 cursor=12:d83876eb:::4:head 2024-03-26T05:24:21.943 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=12 cursor=12:de5d7c5f:::12:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=9 cursor=12:e960b815:::9:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > oid=2 cursor=12:f905c69b:::2:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=1 cursor=12:02547ec2:::1:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:02547ec2:::1:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:02547ec2:::1:head -> 1 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=10 cursor=12:52ea6a34:::10:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:52ea6a34:::10:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:52ea6a34:::10:head -> 10 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=13 cursor=12:566253c9:::13:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:566253c9:::13:head 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:566253c9:::13:head -> 13 2024-03-26T05:24:21.944 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=7 cursor=12:5c6b0b28:::7:head 2024-03-26T05:24:21.945 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:5c6b0b28:::7:head 2024-03-26T05:24:21.945 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:5c6b0b28:::7:head -> 7 2024-03-26T05:24:22.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=14 cursor=12:62a1935d:::14:head 2024-03-26T05:24:22.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:62a1935d:::14:head 2024-03-26T05:24:22.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:62a1935d:::14:head -> 14 2024-03-26T05:24:22.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=0 cursor=12:6cac518f:::0:head 2024-03-26T05:24:22.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:6cac518f:::0:head 2024-03-26T05:24:22.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:6cac518f:::0:head -> 0 2024-03-26T05:24:22.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=15 cursor=12:863748b0:::15:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:863748b0:::15:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:863748b0:::15:head -> 15 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=11 cursor=12:89d3ae78:::11:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:89d3ae78:::11:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:89d3ae78:::11:head -> 11 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=5 cursor=12:b29083e3:::5:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:b29083e3:::5:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:b29083e3:::5:head -> 5 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=8 cursor=12:bd63b0f1:::8:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:bd63b0f1:::8:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:bd63b0f1:::8:head -> 8 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=6 cursor=12:c4fdafeb:::6:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:c4fdafeb:::6:head 2024-03-26T05:24:22.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:c4fdafeb:::6:head -> 6 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=3 cursor=12:cfc208b3:::3:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:cfc208b3:::3:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:cfc208b3:::3:head -> 3 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=4 cursor=12:d83876eb:::4:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:d83876eb:::4:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:d83876eb:::4:head -> 4 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=12 cursor=12:de5d7c5f:::12:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:de5d7c5f:::12:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:de5d7c5f:::12:head -> 12 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=9 cursor=12:e960b815:::9:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:e960b815:::9:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:e960b815:::9:head -> 9 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : oid=2 cursor=12:f905c69b:::2:head 2024-03-26T05:24:22.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:f905c69b:::2:head 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:f905c69b:::2:head -> 2 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:566253c9:::13:head 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:566253c9:::13:head expected=12:566253c9:::13:head 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:566253c9:::13:head -> 13 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=13 expected=13 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:cfc208b3:::3:head 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:cfc208b3:::3:head expected=12:cfc208b3:::3:head 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:cfc208b3:::3:head -> 3 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=3 expected=3 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:c4fdafeb:::6:head 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:c4fdafeb:::6:head expected=12:c4fdafeb:::6:head 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:c4fdafeb:::6:head -> 6 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=6 expected=6 2024-03-26T05:24:22.029 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:e960b815:::9:head 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:e960b815:::9:head expected=12:e960b815:::9:head 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:e960b815:::9:head -> 9 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=9 expected=9 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:de5d7c5f:::12:head 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:de5d7c5f:::12:head expected=12:de5d7c5f:::12:head 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:de5d7c5f:::12:head -> 12 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=12 expected=12 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:d83876eb:::4:head 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:d83876eb:::4:head expected=12:d83876eb:::4:head 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:d83876eb:::4:head -> 4 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=4 expected=4 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:bd63b0f1:::8:head 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:bd63b0f1:::8:head expected=12:bd63b0f1:::8:head 2024-03-26T05:24:22.030 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:bd63b0f1:::8:head -> 8 2024-03-26T05:24:22.031 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=8 expected=8 2024-03-26T05:24:22.031 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:02547ec2:::1:head 2024-03-26T05:24:22.031 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:02547ec2:::1:head expected=12:02547ec2:::1:head 2024-03-26T05:24:22.031 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:02547ec2:::1:head -> 1 2024-03-26T05:24:22.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:24:21 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=1 list_parallel: process_1_[75644]: starting. 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_1_[75644]: creating pool ceph_test_rados_list_parallel.smithi136-75538 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_1_[75644]: created object 0... 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_1_[75644]: created object 25... 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_1_[75644]: created object 49... 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_1_[75644]: finishing. 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_1_[75644]: shutting down. 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_2_[75647]: starting. 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_2_[75647]: listing objects. 2024-03-26T05:24:22.506 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_2_[75647]: listed object 0... 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_2_[75647]: listed object 25... 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_2_[75647]: saw 50 objects 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_2_[75647]: shutting down. 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_3_[76772]: starting. 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_3_[76772]: creating pool ceph_test_rados_list_parallel.smithi136-75538 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_3_[76772]: created object 0... 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_3_[76772]: created object 25... 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_3_[76772]: created object 49... 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_3_[76772]: finishing. 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_3_[76772]: shutting down. 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_4_[76773]: starting. 2024-03-26T05:24:22.507 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_4_[76773]: listing objects. 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_4_[76773]: listed object 0... 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_4_[76773]: listed object 25... 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_4_[76773]: saw 43 objects 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_4_[76773]: shutting down. 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_5_[76774]: starting. 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_5_[76774]: removed 25 objects... 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_5_[76774]: removed half of the objects 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_5_[76774]: removed 50 objects... 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_5_[76774]: removed 50 objects 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_5_[76774]: shutting down. 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_6_[76855]: starting. 2024-03-26T05:24:22.508 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_6_[76855]: creating pool ceph_test_rados_list_parallel.smithi136-75538 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_6_[76855]: created object 0... 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_6_[76855]: created object 25... 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_6_[76855]: created object 49... 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_6_[76855]: finishing. 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_6_[76855]: shutting down. 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_7_[76856]: starting. 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_7_[76856]: listing objects. 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_7_[76856]: listed object 0... 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_7_[76856]: listed object 25... 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_7_[76856]: listed object 50... 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_7_[76856]: saw 56 objects 2024-03-26T05:24:22.509 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_7_[76856]: shutting down. 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_8_[76857]: starting. 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_8_[76857]: added 25 objects... 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_8_[76857]: added half of the objects 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_8_[76857]: added 50 objects... 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_8_[76857]: added 50 objects 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_8_[76857]: shutting down. 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.510 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: Health check failed: 29 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2107963243' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: Health check failed: 29 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2107963243' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T05:24:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:22.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:22.905 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ********************** api_cmd_pp: Running main() from gmock_main.cc 2024-03-26T05:24:22.905 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-26T05:24:22.905 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (428 ms) 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (26 ms) 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (10470 ms) 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (10924 ms total) 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-26T05:24:22.906 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (10924 ms total) 2024-03-26T05:24:22.907 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [----------] Global test environment set-up. 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5065 ms) 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ OK ] LibRadosService.RegisterLate (27 ms) 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: cluster: 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: id: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:24:22.928 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: health: HEALTH_OK 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: services: 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: data: 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: pgs: 132 active+clean 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.929 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: io: 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: cluster: 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: id: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: health: HEALTH_OK 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: services: 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: data: 2024-03-26T05:24:22.930 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: pgs: 132 active+clean 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: io: 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: cluster: 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: id: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: health: HEALTH_WARN 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 29 pool(s) do not have an application enabled 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: services: 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T05:24:22.931 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: data: 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: pools: 36 pools, 1092 pgs 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: pgs: 72.253% pgs unknown 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 15.476% pgs not active 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 789 unknown 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 169 creating+peering 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 134 active+clean 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: io: 2024-03-26T05:24:22.932 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: client: 819 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-26T05:24:22.966 INFO:tasks.workunit.client.0.smithi136.stdout: ********* 2024-03-26T05:24:22.966 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_9_[76912]: starting. 2024-03-26T05:24:22.966 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_9_[76912]: creating pool ceph_test_rados_list_parallel.smithi136-75538 2024-03-26T05:24:22.966 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_9_[76912]: created object 0... 2024-03-26T05:24:22.966 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_9_[76912]: created object 25... 2024-03-26T05:24:22.966 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_9_[76912]: created object 49... 2024-03-26T05:24:22.966 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_9_[76912]: finishing. 2024-03-26T05:24:22.966 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_9_[76912]: shutting down. 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: starting. 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: listing objects. 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: listed object 0... 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: listed object 25... 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: listed object 50... 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: listed object 75... 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: listed object 100... 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: saw 105 objects 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_10_[76913]: shutting down. 2024-03-26T05:24:22.967 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_11_[76914]: starting. 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_11_[76914]: added 25 objects... 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_11_[76914]: added half of the objects 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_11_[76914]: added 50 objects... 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_11_[76914]: added 50 objects 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_11_[76914]: shutting down. 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_12_[76915]: starting. 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_12_[76915]: added 25 objects... 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_12_[76915]: added half of the objects 2024-03-26T05:24:22.968 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_12_[76915]: added 50 objects... 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_12_[76915]: added 50 objects 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_12_[76915]: shutting down. 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_13_[76916]: starting. 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_13_[76916]: removed 25 objects... 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_13_[76916]: removed half of the objects 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_13_[76916]: removed 50 objects... 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_13_[76916]: removed 50 objects 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_13_[76916]: shutting down. 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.969 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_14_[76994]: starting. 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_14_[76994]: creating pool ceph_test_rados_list_parallel.smithi136-75538 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_14_[76994]: created object 0... 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_14_[76994]: created object 25... 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_14_[76994]: created object 49... 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_14_[76994]: finishing. 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_14_[76994]: shutting down. 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: starting. 2024-03-26T05:24:22.970 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: listing objects. 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: Health check failed: 29 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi136-75423-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75484-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2107963243' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75440-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi136-75192-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi136-75071-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi136-75094-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi136-75214-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi136-75172-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi136-75062-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75331-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi136-74875-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:22.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: listed objec handler_error: Running main() from gmock_main.cc 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [----------] Global test environment set-up. 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (11972 ms) 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [----------] 1 test from neocls_handler_error (11972 ms total) 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: 2024-03-26T05:24:24.057 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [----------] Global test environment tear-down 2024-03-26T05:24:24.058 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [==========] 1 test from 1 test suite ran. (11974 ms total) 2024-03-26T05:24:24.058 INFO:tasks.workunit.client.0.smithi136.stdout: handler_error: [ PASSED ] 1 test. 2024-03-26T05:24:24.062 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-26T05:24:24.062 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-26T05:24:24.062 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-26T05:24:24.062 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-26T05:24:24.062 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (291 ms) 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (6 ms) 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (11 ms) 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (28 ms) 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-26T05:24:24.063 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (7 ms) 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (5 ms) 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5 ms) 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-26T05:24:24.064 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (12 ms) 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (10 ms) 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (424 ms total) 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (12076 ms total) 2024-03-26T05:24:24.065 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (12060 ms) 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (12060 ms total) 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (12060 ms total) 2024-03-26T05:24:24.072 INFO:tasks.workunit.client.0.smithi136.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-26T05:24:24.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:24.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:24.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[38106]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T05:24:24.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/749527461' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:24.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:24.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:24.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:24.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[29560]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T05:24:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/749527461' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:24.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi136-75990-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:24.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi136-75924-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:24.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:24 smithi164 ceph-mon[33554]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T05:24:24.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/749527461' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T05:24:24.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:25.350 INFO:tasks.workunit.client.0.smithi136.stdout:t 0... 2024-03-26T05:24:25.350 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: listed object 25... 2024-03-26T05:24:25.350 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: listed object 50... 2024-03-26T05:24:25.350 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: listed object 75... 2024-03-26T05:24:25.350 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: listed object 100... 2024-03-26T05:24:25.350 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: listed object 125... 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: saw 150 objects 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_15_[76995]: shutting down. 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_16_[76996]: starting. 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_16_[76996]: added 25 objects... 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_16_[76996]: added half of the objects 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_16_[76996]: added 50 objects... 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_16_[76996]: added 50 objects 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: process_16_[76996]: shutting down. 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.351 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.352 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.352 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.352 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******************************* 2024-03-26T05:24:25.352 INFO:tasks.workunit.client.0.smithi136.stdout: list_parallel: ******* SUCCESS ********** 2024-03-26T05:24:25.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:25 smithi136 ceph-mon[38106]: pgmap v75: 1060 pgs: 624 creating+peering, 325 active+clean, 111 unknown; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T05:24:25.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:25 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:25.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:25.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:25 smithi136 ceph-mon[29560]: pgmap v75: 1060 pgs: 624 creating+peering, 325 active+clean, 111 unknown; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T05:24:25.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:25 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:25.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:25 smithi164 ceph-mon[33554]: pgmap v75: 1060 pgs: 624 creating+peering, 325 active+clean, 111 unknown; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T05:24:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:25 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:26.630 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-26T05:24:26.630 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-26T05:24:26.630 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-26T05:24:26.630 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-26T05:24:26.630 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-26T05:24:26.630 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (467 ms) 2024-03-26T05:24:26.630 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-26T05:24:26.630 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (19 ms) 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (13915 ms) 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.051172+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi136-75371-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.062679+0000 mon.c [INF] from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.078451+0000 mon.b [INF] from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.214572+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.217237+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.218856+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.220059+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.221255+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.222120+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.222688+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:26.631 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.223351+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:26.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.223907+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:26.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:24.224505+0000 mon.b [INF] from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:26.632 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.309881+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: pgmap v77: 1044 pgs: 544 creating+peering, 309 active+clean, 191 unknown; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 863 B/s wr, 4 op/s 2024-03-26T05:24:26.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:26.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: 14.7 deep-scrub starts 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: 14.7 deep-scrub ok 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: 14.2 deep-scrub starts 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: 14.2 deep-scrub ok 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:26.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/768731903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/165406836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:26.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi136-75192-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:27.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: pgmap v77: 1044 pgs: 544 creating+peering, 309 active+clean, 191 unknown; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 863 B/s wr, 4 op/s 2024-03-26T05:24:27.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:27.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: 14.7 deep-scrub starts 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: 14.7 deep-scrub ok 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: 14.2 deep-scrub starts 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: pgmap v77: 1044 pgs: 544 creating+peering, 309 active+clean, 191 unknown; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 863 B/s wr, 4 op/s 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:27.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: 14.7 deep-scrub starts 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: 14.7 deep-scrub ok 2024-03-26T05:24:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: 14.2 deep-scrub starts 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: 14.2 deep-scrub ok 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/768731903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/165406836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:27.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi136-75192-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: 14.2 deep-scrub ok 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:27.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/768731903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/165406836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:27.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi136-75192-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:27.573 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:24:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:27.637 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.312622+0000 mon.a [INF] from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:27.637 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.318794+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.637 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.326648+0000 mon.b [INF] from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:27.637 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.334852+0000 mon.b [INF] from='client.? 172.21.15.136:0/768731903' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.637 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.337279+0000 mon.b [INF] from='client.? 172.21.15.136:0/165406836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.637 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.341932+0000 mon.b [INF] from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:27.637 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.422008+0000 mon.b [INF] from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:25.433179+0000 mon.b [INF] from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.088697+0000 mon.a [INF] from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.093128+0000 mon.a [INF] from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.093750+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.094321+0000 mon.a [INF] from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi136-75192-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.096034+0000 mon.a [INF] from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.096262+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.101425+0000 mon.b [INF] from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.341411+0000 mon.a [INF] from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:27.638 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.341612+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.8 deep-scrub starts 2024-03-26T05:24:27.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.8 deep-scrub ok 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.0 deep-scrub starts 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.0 deep-scrub ok 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.1 deep-scrub starts 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.1 deep-scrub ok 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.6 deep-scrub starts 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.6 deep-scrub ok 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.5 deep-scrub starts 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: 14.5 deep-scrub ok 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi136-75192-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2770602279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi136-75192-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi136-74810-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:27.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:27.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.8 deep-scrub starts 2024-03-26T05:24:28.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.8 deep-scrub ok 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.0 deep-scrub starts 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.0 deep-scrub ok 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.1 deep-scrub starts 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.1 deep-scrub ok 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.6 deep-scrub starts 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.6 deep-scrub ok 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.5 deep-scrub starts 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: 14.5 deep-scrub ok 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi136-75192-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2770602279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi136-75192-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi136-74810-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.8 deep-scrub starts 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.8 deep-scrub ok 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.0 deep-scrub starts 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.0 deep-scrub ok 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.1 deep-scrub starts 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.1 deep-scrub ok 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.6 deep-scrub starts 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.6 deep-scrub ok 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:28.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.5 deep-scrub starts 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: 14.5 deep-scrub ok 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi136-75192-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2770602279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T05:24:28.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi136-75192-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:28.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi136-74810-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:28.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:28.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:28.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.658 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.341991+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:28.658 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.342740+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.658 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.342830+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.658 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.342914+0000 mon.a [INF] from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi136-75192-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:28.658 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.343002+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi136-74761-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.658 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.349317+0000 mon.c [INF] from='client.? 172.21.15.136:0/2770602279' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.658 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.420017+0000 mon.b [INF] from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.638141+0000 mon.a [INF] from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi136-75192-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.644841+0000 mon.a [INF] from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi136-74810-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.645805+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.650182+0000 mon.c [INF] from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.368206+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.368433+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:26.644322+0000 client.admin [INF] onexx 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.368510+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.368560+0000 mon.a [INF] from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi136-74810-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.368638+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.374421+0000 mon.b [INF] from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.379351+0000 mon.c [INF] from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:28.659 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.642839+0000 mon.c [INF] from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.643396+0000 mon.b [INF] from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.652710+0000 mon.a [INF] from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi136-74810-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.654239+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.657092+0000 mon.b [INF] from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.659492+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.660226+0000 mon.b [INF] from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.903513+0000 mon.a [WRN] Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.905515+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.907573+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.907929+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.908217+0000 mon.c [INF] from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:27.908622+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.660 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:28.387222+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: 14.9 deep-scrub starts 2024-03-26T05:24:28.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: 14.9 deep-scrub ok 2024-03-26T05:24:28.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: 14.4 deep-scrub starts 2024-03-26T05:24:28.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: 14.4 deep-scrub ok 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: onexx 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: 14.3 deep-scrub starts 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: 14.3 deep-scrub ok 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi136-74810-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi136-74810-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:28.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi136-75192-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:29.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: 14.9 deep-scrub starts 2024-03-26T05:24:29.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: 14.9 deep-scrub ok 2024-03-26T05:24:29.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: 14.4 deep-scrub starts 2024-03-26T05:24:29.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: 14.4 deep-scrub ok 2024-03-26T05:24:29.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: onexx 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: 14.3 deep-scrub starts 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: 14.3 deep-scrub ok 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi136-74810-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi136-74810-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:29.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:29.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi136-75192-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: 14.9 deep-scrub starts 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: 14.9 deep-scrub ok 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: 14.4 deep-scrub starts 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: 14.4 deep-scrub ok 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: onexx 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: 14.3 deep-scrub starts 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: 14.3 deep-scrub ok 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi136-74810-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:29.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/967411579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi136-74810-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:29.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:29.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:29.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:29.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:29.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:29.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi136-75192-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:29.415 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:28.388192+0000 mon.a [INF] from='client.? 172. cls: Running main() from gmock_main.cc 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [----------] Global test environment set-up. 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [ OK ] NeoRadosCls.DNE (9901 ms) 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (7446 ms) 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [----------] 2 tests from NeoRadosCls (17347 ms total) 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [----------] Global test environment tear-down 2024-03-26T05:24:29.416 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [==========] 2 tests from 1 test suite ran. (17353 ms total) 2024-03-26T05:24:29.417 INFO:tasks.workunit.client.0.smithi136.stdout: cls: [ PASSED ] 2 tests. 2024-03-26T05:24:29.429 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_1_[75698]: starting. 2024-03-26T05:24:29.429 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_1_[75698]: creating pool ceph_test_rados_delete_pools_parallel.smithi136-75657 2024-03-26T05:24:29.429 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_1_[75698]: created object 0... 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_1_[75698]: created object 25... 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_1_[75698]: created object 49... 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_1_[75698]: finishing. 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_1_[75698]: shutting down. 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_2_[75700]: starting. 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_2_[75700]: deleting pool ceph_test_rados_delete_pools_parallel.smithi136-75657 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_2_[75700]: shutting down. 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: ******************************* 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_3_[77227]: starting. 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_3_[77227]: creating pool ceph_test_rados_delete_pools_parallel.smithi136-75657 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_3_[77227]: created object 0... 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_3_[77227]: created object 25... 2024-03-26T05:24:29.430 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_3_[77227]: created object 49... 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_3_[77227]: finishing. 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_3_[77227]: shutting down. 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: ******************************* 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_5_[77229]: starting. 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_5_[77229]: listing objects. 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_5_[77229]: listed object 0... 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_5_[77229]: listed object 25... 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_5_[77229]: saw 50 objects 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_5_[77229]: shutting down. 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: ******************************* 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_4_[77228]: starting. 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_4_[77228]: deleting pool ceph_test_rados_delete_pools_parallel.smithi136-75657 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: process_4_[77228]: shutting down. 2024-03-26T05:24:29.431 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: ******************************* 2024-03-26T05:24:29.432 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: ******************************* 2024-03-26T05:24:29.432 INFO:tasks.workunit.client.0.smithi136.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-26T05:24:30.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: pgmap v80: 852 pgs: 240 creating+peering, 556 active+clean, 56 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 33 MiB/s wr, 489 op/s 2024-03-26T05:24:30.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: pgmap v83: 876 pgs: 191 creating+peering, 388 active+clean, 297 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 542 op/s 2024-03-26T05:24:30.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:30.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: twoxx 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: pgmap v80: 852 pgs: 240 creating+peering, 556 active+clean, 56 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 33 MiB/s wr, 489 op/s 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: pgmap v83: 876 pgs: 191 creating+peering, 388 active+clean, 297 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 542 op/s 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T05:24:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: twoxx 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi136-74810-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T05:24:30.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi136-74810-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:30.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: pgmap v80: 852 pgs: 240 creating+peering, 556 active+clean, 56 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 33 MiB/s wr, 489 op/s 2024-03-26T05:24:30.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: pgmap v83: 876 pgs: 191 creating+peering, 388 active+clean, 297 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 542 op/s 2024-03-26T05:24:30.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi136-74781-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2590932453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4231269672' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]: dispatch 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: twoxx 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T05:24:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi136-74810-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:30.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi136-75990-1"}]': finished 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi136-74781-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi136-75924-1"}]': finished 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:30.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:30.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:30.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:31.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T05:24:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T05:24:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:31.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_1_[75688]: starting. 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_1_[75688]: creating pool ceph_test_rados_open_pools_parallel.smithi136-75635 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_1_[75688]: created object 0... 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_1_[75688]: created object 25... 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_1_[75688]: created object 49... 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_1_[75688]: finishing. 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_1_[75688]: shutting down. 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_2_[75692]: starting. 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_2_[75692]: rados_pool_create. 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_2_[75692]: rados_ioctx_create. 2024-03-26T05:24:31.533 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_2_[75692]: shutting down. 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: ******************************* 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_3_[77230]: starting. 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_3_[77230]: creating pool ceph_test_rados_open_pools_parallel.smithi136-75635 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_3_[77230]: created object 0... 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_3_[77230]: created object 25... 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_3_[77230]: created object 49... 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_3_[77230]: finishing. 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_3_[77230]: shutting down. 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: ******************************* 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_4_[77231]: starting. 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_4_[77231]: rados_pool_create. 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_4_[77231]: rados_ioctx_create. 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: process_4_[77231]: shutting down. 2024-03-26T05:24:31.534 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: ******************************* 2024-03-26T05:24:31.535 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: ******************************* 2024-03-26T05:24:31.535 INFO:tasks.workunit.client.0.smithi136.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: Running main() from gmock_main.cc 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [----------] Global test environment set-up. 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9233 ms) 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (3958 ms) 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (6256 ms) 2024-03-26T05:24:31.561 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [----------] 3 tests from NeoRadosCmd (19447 ms total) 2024-03-26T05:24:31.562 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: 2024-03-26T05:24:31.562 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [----------] Global test environment tear-down 2024-03-26T05:24:31.562 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [==========] 3 tests from 1 test suite ran. (19448 ms total) 2024-03-26T05:24:31.562 INFO:tasks.workunit.client.0.smithi136.stdout: cmd: [ PASSED ] 3 tests. 2024-03-26T05:24:32.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:24:31 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:24:32.746 INFO:tasks.workunit.client.0.smithi136.stdout:21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi136-75192-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:32.746 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:29.617312+0000 mon.c [INF] from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:32.746 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.001836+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T05:24:32.746 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.495389+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:32.746 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.495466+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi136-74915-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:32.746 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.495516+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:32.746 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.500330+0000 mon.c [INF] from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.503071+0000 mon.b [INF] from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.495560+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.495601+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.495645+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.503534+0000 mon.b [INF] from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.504123+0000 mon.b [INF] from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.504466+0000 mon.b [INF] from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:30.504877+0000 mon.b [INF] from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.012824+0000 mon.b [INF] from='client.? 172.21.15.136:0/1062791565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.012901+0000 mon.c [INF] from='client.? 172.21.15.136:0/2520357668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.017310+0000 mon.a [INF] from='client.? 172.21.15.136:0/3467898186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.018731+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.019766+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:32.747 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.027429+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:32.748 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.274684+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:32.748 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.274928+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:32.748 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.275108+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:32.748 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.275326+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:32.748 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.275516+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:32.748 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.517541+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:32.748 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.518713+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:32.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.518813+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_ watch_notify: Running main() from gmock_main.cc 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: handle_notify cookie 94514073428320 notify_id 335007449089 notifier_gid 14844 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (10701 ms) 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: Trying... 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: handle_notify cookie 94514086228704 notify_id 356482285569 notifier_gid 25018 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: Waiting for 3.000000000s 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: Timed out. 2024-03-26T05:24:32.772 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: Flushing... 2024-03-26T05:24:32.773 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: Flushed... 2024-03-26T05:24:32.773 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (9869 ms) 2024-03-26T05:24:32.773 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (20570 ms total) 2024-03-26T05:24:32.773 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: 2024-03-26T05:24:32.773 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-26T05:24:32.773 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (20570 ms total) 2024-03-26T05:24:32.773 INFO:tasks.workunit.client.0.smithi136.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-26T05:24:33.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: pgmap v86: 788 pgs: 86 creating+peering, 388 active+clean, 314 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:24:33.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:33.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1062791565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2520357668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3467898186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: pgmap v86: 788 pgs: 86 creating+peering, 388 active+clean, 314 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1062791565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2520357668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3467898186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:33.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:33.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:33.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: pgmap v86: 788 pgs: 86 creating+peering, 388 active+clean, 314 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi136-75990-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi136-75924-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2090235626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1062791565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2520357668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3467898186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi136-75172-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi136-74875-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:33.748 INFO:tasks.workunit.client.0.smithi136.stdout:type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:33.748 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.518903+0000 mon.a [INF] from='client.? 172.21.15.136:0/3467898186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.518979+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.519060+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.519139+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.519217+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.524729+0000 mon.b [INF] from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.555214+0000 mon.c [INF] from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2"}]: dispatch 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:31.557535+0000 mon.c [INF] from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.006979+0000 mon.b [INF] from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.007046+0000 client.admin [INF] threexx 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.501947+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.509224+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.510476+0000 mon.a [INF] from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.510635+0000 mon.a [INF] from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.759701+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.761743+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.761843+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.761920+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.762008+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.762088+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.762149+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.762219+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.762328+0000 mon.a [INF] from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.762395+0000 mon.a [INF] from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.767939+0000 mon.c [INF] from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.771474+0000 mon.a [INF] from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.771726+0000 mon.a [INF] from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:33.750 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.772289+0000 mon.c [INF] from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T05:24:33.751 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.773649+0000 mon.c [INF] from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:24:33.751 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:32.836501+0000 mon.b [INF] from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]: dispatch 2024-03-26T05:24:33.751 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:33.031221+0000 mon.a [WRN] Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:33.032428+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-753 api_io: Running main() from gmock_main.cc 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [----------] Global test environment set-up. 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (1266 ms) 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: no timeout :/ 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: no timeout :/ 2024-03-26T05:24:33.788 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: no timeout :/ 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: no timeout :/ 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: no timeout :/ 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (60 ms) 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (6 ms) 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.Checksum (6 ms) 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (8 ms) 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (8 ms) 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (9 ms) 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (4 ms) 2024-03-26T05:24:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (4 ms) 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.RmXattr (18 ms) 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [----------] 14 tests from LibRadosIo (1404 ms total) 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-26T05:24:33.790 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1116 ms) 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (8 ms) 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (6 ms) 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8 ms) 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-03-26T05:24:33.791 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-26T05:24:33.792 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (7 ms) 2024-03-26T05:24:33.792 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-26T05:24:33.792 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-26T05:24:33.792 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1165 ms total) 2024-03-26T05:24:33.792 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: 2024-03-26T05:24:33.792 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [----------] Global test environment tear-down 2024-03-26T05:24:33.792 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [==========] 24 tests from 2 test suites ran. (22023 ms total) 2024-03-26T05:24:33.792 INFO:tasks.workunit.client.0.smithi136.stdout: api_io: [ PASSED ] 24 tests. 2024-03-26T05:24:33.811 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: Running main() from gmock_main.cc 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [----------] Global test environment set-up. 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStat.Stat (249 ms) 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (45 ms) 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStat.StatNS (15 ms) 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3 ms) 2024-03-26T05:24:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [----------] 5 tests from LibRadosStat (312 ms total) 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (923 ms) 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (9 ms) 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (398 ms) 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (244 ms) 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1574 ms total) 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [----------] Global test environment tear-down 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (21899 ms total) 2024-03-26T05:24:33.813 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-26T05:24:33.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-26T05:24:33.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-26T05:24:33.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-26T05:24:33.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-26T05:24:33.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: seed 74824 2024-03-26T05:24:33.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (101 ms) 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (10 ms) 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (1145 ms) 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (18 ms) 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (20 ms) 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (6 ms) 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-26T05:24:33.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (5 ms) 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (6 ms) 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (8 ms) 2024-03-26T05:24:33.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (25 ms) 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (8 ms) 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (6 ms) 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (6 ms) 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (1396 ms total) 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-26T05:24:33.965 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1172 ms) 2024-03-26T05:24:33.966 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-26T05:24:33.966 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (18 ms) 2024-03-26T05:24:33.966 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-26T05:24:33.966 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-03-26T05:24:33.966 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-26T05:24:33.966 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (5 ms) 2024-03-26T05:24:33.966 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-26T05:24:33.966 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (4 ms) 2024-03-26T05:24:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3467898186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2"}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: pgmap v89: 820 pgs: 89 creating+peering, 612 active+clean, 119 unknown; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: threexx 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3467898186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2"}]: dispatch 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: pgmap v89: 820 pgs: 89 creating+peering, 612 active+clean, 119 unknown; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-03-26T05:24:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: threexx 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:34.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]': finished 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T05:24:34.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]: dispatch 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]: dispatch 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]': finished 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T05:24:34.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]: dispatch 2024-03-26T05:24:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3467898186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi136-74761-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-4", "overlaypool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi136-75371-1"}]': finished 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2"}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: pgmap v89: 820 pgs: 89 creating+peering, 612 active+clean, 119 unknown; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: threexx 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi136-75990-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi136-74915-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi136-74824-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi136-75924-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-4-cache", "mode": "writeback"}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi136-74929-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]: dispatch 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]: dispatch 2024-03-26T05:24:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T05:24:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:24:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]: dispatch 2024-03-26T05:24:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:24:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T05:24:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]: dispatch 2024-03-26T05:24:35.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T05:24:35.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T05:24:35.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: fourxx 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]': finished 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]: dispatch 2024-03-26T05:24:35.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:35.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T05:24:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T05:24:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: fourxx 2024-03-26T05:24:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T05:24:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:35.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]': finished 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/571365897' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: fourxx 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/406327024' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi136-75192-7"}]': finished 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3215352246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi136-74810-16"}]': finished 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]': finished 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3007824497' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]: dispatch 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]: dispatch 2024-03-26T05:24:35.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:35.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN list: Running main() from gmock_main.cc 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [----------] Global test environment set-up. 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [----------] 3 tests from NeoradosList 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [ OK ] NeoradosList.ListObjects (11907 ms) 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [ OK ] NeoradosList.ListObjectsNS (5405 ms) 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [ OK ] NeoradosList.ListObjectsMany (6428 ms) 2024-03-26T05:24:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: list: [----------] 3 tests from NeoradosList (23740 ms total) 2024-03-26T05:24:35.890 INFO:tasks.workunit.client.0.smithi136.stdout: list: 2024-03-26T05:24:35.890 INFO:tasks.workunit.client.0.smithi136.stdout: list: [----------] Global test environment tear-down 2024-03-26T05:24:35.890 INFO:tasks.workunit.client.0.smithi136.stdout: list: [==========] 3 tests from 1 test suite ran. (23740 ms total) 2024-03-26T05:24:35.890 INFO:tasks.workunit.client.0.smithi136.stdout: list: [ PASSED ] 3 tests. 2024-03-26T05:24:36.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: pgmap v92: 692 pgs: 516 active+clean, 176 unknown; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:36.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:36.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T05:24:36.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:36.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T05:24:36.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:24:36.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2168511631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T05:24:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]: dispatch 2024-03-26T05:24:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:24:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:36.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: pgmap v92: 692 pgs: 516 active+clean, 176 unknown; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:36.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:36.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T05:24:36.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:36.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T05:24:36.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2168511631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: pgmap v92: 692 pgs: 516 active+clean, 176 unknown; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T05:24:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-4", "tierpool": "test-rados-api-smithi136-75071-4-cache"}]': finished 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1956038666' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2168511631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]: dispatch 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:36.878 INFO:tasks.workunit.client.0.smithi136.stdout:71-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:36.878 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:33.270022+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:24:36.879 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:33.274088+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T05:24:36.879 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: got: 2024-03-26T05:24:33.275936+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-4"}]: dispatch 2024-03-26T05:24:36.879 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (10534 ms) 2024-03-26T05:24:36.879 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (24935 ms total) 2024-03-26T05:24:36.879 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: 2024-03-26T05:24:36.879 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-26T05:24:36.879 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (24935 ms total) 2024-03-26T05:24:36.879 INFO:tasks.workunit.client.0.smithi136.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-26T05:24:37.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:24:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:37.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]': finished 2024-03-26T05:24:37.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]': finished 2024-03-26T05:24:37.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1787763439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4228017730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]': finished 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]': finished 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:37.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1787763439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4228017730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:37.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:37.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:37.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:37.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75371-6", "pg_num": 4}]': finished 2024-03-26T05:24:37.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi136-74929-1","app":"app1","key":"key1"}]': finished 2024-03-26T05:24:37.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:37.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1787763439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4228017730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:37.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:37.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:37.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:37.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:37.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:37.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: Running main() from gmock_main.cc 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [----------] Global test environment set-up. 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (17268 ms) 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (8666 ms) 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (25934 ms total) 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: 2024-03-26T05:24:38.078 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [----------] Global test environment tear-down 2024-03-26T05:24:38.079 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (25934 ms total) 2024-03-26T05:24:38.079 INFO:tasks.workunit.client.0.smithi136.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: seed 74875 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (1339 ms) 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (3 ms) 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1026 ms) 2024-03-26T05:24:38.087 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1019 ms) 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3 ms) 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4 ms) 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (2 ms) 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (3400 ms total) 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1296 ms) 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-26T05:24:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (3 ms) 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1086 ms) 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1016 ms) 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (6 ms) 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (4 ms) 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3418 ms total) 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-26T05:24:38.089 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (26300 ms total) 2024-03-26T05:24:38.090 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-26T05:24:38.090 INFO:tasks.workunit.client.0.smithi136.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-26T05:24:38.090 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (6 ms) 2024-03-26T05:24:38.090 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-26T05:24:38.090 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-03-26T05:24:38.090 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2 ms) 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-03-26T05:24:38.091 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9 ms) 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (7 ms) 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1268 ms total) 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (26322 ms total) 2024-03-26T05:24:38.092 INFO:tasks.workunit.client.0.smithi136.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: Running main() from gmock_main.cc 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [----------] Global test environment set-up. 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ OK ] NeoRadosPools.PoolList (9508 ms) 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4681 ms) 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2322 ms) 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-26T05:24:38.123 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5109 ms) 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2138 ms) 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2188 ms) 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [----------] 6 tests from NeoRadosPools (25946 ms total) 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [----------] Global test environment tear-down 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [==========] 6 tests from 1 test suite ran. (25946 ms total) 2024-03-26T05:24:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: pool: [ PASSED ] 6 tests. 2024-03-26T05:24:38.132 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-26T05:24:38.132 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-26T05:24:38.132 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-26T05:24:38.132 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-26T05:24:38.132 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: seed 75172 2024-03-26T05:24:38.132 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-26T05:24:38.132 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (260 ms) 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (15 ms) 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1 ms) 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (23 ms) 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (300 ms total) 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1303 ms) 2024-03-26T05:24:38.133 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1097 ms) 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (27 ms) 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (2428 ms total) 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (26233 ms total) 2024-03-26T05:24:38.134 INFO:tasks.workunit.client.0.smithi136.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-26T05:24:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: pgmap v94: 820 pgs: 128 creating+peering, 692 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-03-26T05:24:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]': finished 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:38.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]: dispatch 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]: dispatch 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:38.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2579839741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:38.830 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: pgmap v94: 820 pgs: 128 creating+peering, 692 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-03-26T05:24:38.830 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:38.830 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:38.830 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:38.830 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:38.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]': finished 2024-03-26T05:24:38.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]: dispatch 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]: dispatch 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:38.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2579839741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: pgmap v94: 820 pgs: 128 creating+peering, 692 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-03-26T05:24:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi136-74761-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/216109288' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2218749442' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1864999320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2585783106' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2929798320' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]: dispatch 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]: dispatch 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]: dispatch 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]: dispatch 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]: dispatch 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi136-75214-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi136-75990-2"}]': finished 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "overlaypool": "test-rados-api-smithi136-75371-6"}]': finished 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi136-74875-10"}]': finished 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi136-75172-7"}]': finished 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi136-74824-22"}]': finished 2024-03-26T05:24:38.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi136-75924-2"}]': finished 2024-03-26T05:24:38.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T05:24:38.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]: dispatch 2024-03-26T05:24:38.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:38.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]: dispatch 2024-03-26T05:24:38.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:38.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:38.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2579839741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:39.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:39 smithi136 ceph-mon[38106]: pgmap v97: 688 pgs: 164 unknown, 32 creating+peering, 492 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.6 KiB/s wr, 7 op/s 2024-03-26T05:24:39.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:39.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:39.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:39.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:39 smithi136 ceph-mon[29560]: pgmap v97: 688 pgs: 164 unknown, 32 creating+peering, 492 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.6 KiB/s wr, 7 op/s 2024-03-26T05:24:39.736 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:39.736 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:39.736 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:39.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:39 smithi164 ceph-mon[33554]: pgmap v97: 688 pgs: 164 unknown, 32 creating+peering, 492 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.6 KiB/s wr, 7 op/s 2024-03-26T05:24:39.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:39.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:39.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:40.804 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: Running main() from gmock_main.cc 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [----------] Global test environment set-up. 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (1241 ms) 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLock.LockShared (61 ms) 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1015 ms) 2024-03-26T05:24:40.805 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1030 ms) 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (47 ms) 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLock.Unlock (13 ms) 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4 ms) 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [----------] 8 tests from LibRadosLock (3414 ms total) 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: 2024-03-26T05:24:40.806 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1167 ms) 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (10 ms) 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1007 ms) 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1070 ms) 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (40 ms) 2024-03-26T05:24:40.807 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (7 ms) 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (24 ms) 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (11 ms) 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3336 ms total) 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [----------] Global test environment tear-down 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (28981 ms total) 2024-03-26T05:24:40.808 INFO:tasks.workunit.client.0.smithi136.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-26T05:24:41.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:41.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]': finished 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2579839741' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3793483285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3330289701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1100864998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:41.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:41.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]': finished 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2579839741' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3793483285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3330289701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1100864998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75371-6", "mode": "writeback"}]': finished 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2579839741' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi136-75990-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2498690312' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3793483285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3330289701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1100864998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:41.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:42.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:24:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:24:42.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: pgmap v100: 624 pgs: 204 unknown, 420 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:24:42.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:42.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T05:24:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:42 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:42.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: pgmap v100: 624 pgs: 204 unknown, 420 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:24:42.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:42.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: pgmap v100: 624 pgs: 204 unknown, 420 active+clean; 145 MiB data, 795 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi136-74915-10"}]': finished 2024-03-26T05:24:42.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi136-75062-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi136-74761-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T05:24:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:42 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:42.961 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-26T05:24:42.961 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T05:24:42.961 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-26T05:24:42.961 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-26T05:24:42.961 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: notify 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (2409 ms) 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (1013 ms) 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (3422 ms total) 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: notify 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (141 ms) 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: notify 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5839 ms) 2024-03-26T05:24:42.962 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (28 ms) 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (13 ms) 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930180896 notify_id 403726925826 notifier_gid 25313 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (26 ms) 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930180896 notify_id 403726925831 notifier_gid 25313 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (18 ms) 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930180896 notify_id 403726925832 notifier_gid 25313 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (22 ms) 2024-03-26T05:24:42.963 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930180896 notify_id 403726925833 notifier_gid 25313 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (20 ms) 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930185856 notify_id 403726925826 notifier_gid 25313 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (17 ms) 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930204976 notify_id 403726925825 notifier_gid 25313 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (20 ms) 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: trying... 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930204976 notify_id 403726925827 notifier_gid 25313 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: timed out 2024-03-26T05:24:42.964 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: flushing 2024-03-26T05:24:43.017 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:43.017 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:43.017 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5913 ms) 2024-03-26T05:24:43.017 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-26T05:24:43.018 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ OK ] LibRadosService.Status (20045 ms) 2024-03-26T05:24:43.018 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [----------] 4 tests from LibRadosService (31050 ms total) 2024-03-26T05:24:43.018 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: 2024-03-26T05:24:43.018 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [----------] Global test environment tear-down 2024-03-26T05:24:43.018 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31050 ms total) 2024-03-26T05:24:43.018 INFO:tasks.workunit.client.0.smithi136.stdout: api_service: [ PASSED ] 4 tests. 2024-03-26T05:24:43.020 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-26T05:24:43.020 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-26T05:24:43.020 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-26T05:24:43.020 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-26T05:24:43.020 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5036 ms) 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (807 ms) 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20051 ms) 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: attempt 0 of 20 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: failed to deregister: 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: { 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "epoch": 19, 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "modified": "2024-03-26T05:24:39.858507+0000", 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "services": { 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "laundry": { 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "daemons": { 2024-03-26T05:24:43.021 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "summary": "", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "close-test-pid75462": { 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_epoch": 19, 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_stamp": "2024-03-26T05:24:37.974581+0000", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "gid": 26765, 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "addr": "172.21.15.136:0/4021074576", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "metadata": { 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro": "centos", 2024-03-26T05:24:43.022 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_version": "9", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "foo": "bar", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "hostname": "smithi136", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_total_kb": "32488364", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "os": "Linux", 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "this": "that" 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: }, 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "task_status": {} 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: }, 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "pid75300": { 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_epoch": 18, 2024-03-26T05:24:43.023 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_stamp": "2024-03-26T05:24:22.992162+0000", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "gid": 25027, 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "addr": "172.21.15.136:0/1124436779", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "metadata": { 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro": "centos", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_version": "9", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "foo": "bar", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "hostname": "smithi136", 2024-03-26T05:24:43.024 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_total_kb": "32488364", 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "os": "Linux", 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "this": "that" 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: }, 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "task_status": {} 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: }, 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "rgw": { 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "daemons": { 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "summary": "", 2024-03-26T05:24:43.025 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "24340": { 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_epoch": 12, 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_stamp": "2024-03-26T05:21:37.956799+0000", 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "gid": 24340, 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "addr": "172.21.15.136:0/2659226996", 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "metadata": { 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "container_hostname": "smithi136", 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T05:24:43.026 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro": "centos", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_version": "8", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "frontend_type#0": "beast", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "hostname": "smithi136", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "id": "foo.a", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_total_kb": "32488364", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "num_handles": "1", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "os": "Linux", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "pid": "2", 2024-03-26T05:24:43.027 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "realm_id": "", 2024-03-26T05:24:43.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "realm_name": "", 2024-03-26T05:24:43.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "zone_id": "a8944c22-3dfd-4a69-a2c5-4fdbdde5a514", 2024-03-26T05:24:43.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "zone_name": "default", 2024-03-26T05:24:43.028 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "zonegroup_id": "41faacb0-282b-439f-967a-acc7f76dac51", 2024-03-26T05:24:43.210 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "zonegroup_name": "d api_watch_notify: Running main() from gmock_main.cc 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (275 ms) 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94277939033312 err -107 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (7 ms) 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-26T05:24:43.211 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94277939033312 err -107 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1032 ms) 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_cb from 24649 notify_id 339302416384 cookie 94277939059840 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (34 ms) 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_cb from 24649 notify_id 339302416384 cookie 94277939056208 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (7 ms) 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_cb from 24649 notify_id 339302416384 cookie 94277939805280 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (6 ms) 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_cb from 24649 notify_id 339302416385 cookie 94277939788464 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_cb from 24649 notify_id 339302416385 cookie 94277939805216 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (8 ms) 2024-03-26T05:24:43.212 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_cb from 24649 notify_id 339302416386 cookie 94277939805216 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_cb from 24649 notify_id 339302416387 cookie 94277939805216 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3005 ms) 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94277939805216 err -107 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_cb from 24649 notify_id 369367187461 cookie 94277939805216 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94277939805216 err -107 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10385 ms total) 2024-03-26T05:24:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: 2024-03-26T05:24:43.214 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-26T05:24:43.214 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-26T05:24:43.214 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T05:24:43.214 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1672 ms) 2024-03-26T05:24:43.214 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1672 ms total) 2024-03-26T05:24:43.214 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: 2024-03-26T05:24:43.214 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-26T05:24:43.214 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (31283 ms total) 2024-03-26T05:24:43.217 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-26T05:24:44.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: pgmap v102: 648 pgs: 57 creating+peering, 51 creating+activating, 2 active+clean+snaptrim, 39 unknown, 499 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 5.5 KiB/s wr, 2 op/s 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3609490969' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3343577327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1909872715' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2450103523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: pgmap v102: 648 pgs: 57 creating+peering, 51 creating+activating, 2 active+clean+snaptrim, 39 unknown, 499 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 5.5 KiB/s wr, 2 op/s 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:44.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3609490969' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3343577327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1909872715' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2450103523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:44 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:44.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: pgmap v102: 648 pgs: 57 creating+peering, 51 creating+activating, 2 active+clean+snaptrim, 39 unknown, 499 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 5.5 KiB/s wr, 2 op/s 2024-03-26T05:24:44.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:44.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi136-75990-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2515120291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3609490969' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi136-75214-12"}]': finished 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3343577327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1909872715' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2450103523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:44.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:44.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:44 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: pgmap v105: 688 pgs: 38 creating+peering, 26 creating+activating, 194 unknown, 430 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 5.8 KiB/s wr, 5 op/s 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:45.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2050474555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: pgmap v105: 688 pgs: 38 creating+peering, 26 creating+activating, 194 unknown, 430 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 5.8 KiB/s wr, 5 op/s 2024-03-26T05:24:45.338 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2050474555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:45.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:45 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: pgmap v105: 688 pgs: 38 creating+peering, 26 creating+activating, 194 unknown, 430 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 5.8 KiB/s wr, 5 op/s 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi136-74761-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:45.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T05:24:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2050474555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:45 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]': finished 2024-03-26T05:24:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[29560]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T05:24:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]': finished 2024-03-26T05:24:46.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:46.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[38106]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T05:24:46.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:46.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:46.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-7"}]': finished 2024-03-26T05:24:46.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi136-75094-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:46.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:46 smithi164 ceph-mon[33554]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T05:24:46.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:47.526 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: pgmap v108: 624 pgs: 51 creating+peering, 7 creating+activating, 45 unknown, 521 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/607637571' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:47.527 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:24:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: pgmap v108: 624 pgs: 51 creating+peering, 7 creating+activating, 45 unknown, 521 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:24:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T05:24:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/607637571' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:24:47.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:24:47.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:47.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: pgmap v108: 624 pgs: 51 creating+peering, 7 creating+activating, 45 unknown, 521 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:24:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T05:24:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/607637571' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:24:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:24:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:24:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:48.173 INFO:tasks.workunit.client.0.smithi136.stdout:efault" 2024-03-26T05:24:48.173 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: }, 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "task_status": {} 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: attempt 1 of 20 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: failed to deregister: 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: { 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "epoch": 21, 2024-03-26T05:24:48.174 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "modified": "2024-03-26T05:24:43.861622+0000", 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "services": { 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "laundry": { 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "daemons": { 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "summary": "", 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "close-test-pid75462": { 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_epoch": 21, 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_stamp": "2024-03-26T05:24:43.107386+0000", 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "gid": 31225, 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "addr": "172.21.15.136:0/273768536", 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "metadata": { 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T05:24:48.175 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro": "centos", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_version": "9", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "foo": "bar", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "hostname": "smithi136", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_total_kb": "32488364", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "os": "Linux", 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "this": "that" 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: }, 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "task_status": {} 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:48.176 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: }, 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "rgw": { 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "daemons": { 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "summary": "", 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "24340": { 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_epoch": 12, 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "start_stamp": "2024-03-26T05:21:37.956799+0000", 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "gid": 24340, 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "addr": "172.21.15.136:0/2659226996", 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "metadata": { 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T05:24:48.177 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "container_hostname": "smithi136", 2024-03-26T05:24:49.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:24:49.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:49.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2638618361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3907054243' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: pool 'PoolQuotaPP_smithi136-74781-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]': finished 2024-03-26T05:24:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T05:24:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2638618361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3907054243' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: pool 'PoolQuotaPP_smithi136-74781-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]': finished 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T05:24:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi136-74761-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2638618361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3907054243' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: pool 'PoolQuotaPP_smithi136-74781-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T05:24:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:24:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-7"}]': finished 2024-03-26T05:24:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75403-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T05:24:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2222313689' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:49.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4 ec_list: Running main() from gmock_main.cc 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [----------] Global test environment set-up. 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (17249 ms) 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (8671 ms) 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (11335 ms) 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [----------] 3 tests from NeoradosECList (37255 ms total) 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [----------] Global test environment tear-down 2024-03-26T05:24:49.448 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (37255 ms total) 2024-03-26T05:24:49.449 INFO:tasks.workunit.client.0.smithi136.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-26T05:24:50.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: pgmap v111: 648 pgs: 96 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2024-03-26T05:24:50.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:50.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:50.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:50.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:50.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T05:24:50.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2737631671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:50.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:50.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: pgmap v111: 648 pgs: 96 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2024-03-26T05:24:50.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:50.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:50.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:50.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:50.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2737631671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: pgmap v111: 648 pgs: 96 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]: dispatch 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi136-75990-3"}]': finished 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2737631671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:50.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: Running main() from gmock_main.cc 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [----------] Global test environment set-up. 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (11884 ms) 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (5352 ms) 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (4357 ms) 2024-03-26T05:24:51.679 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3566 ms) 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3818 ms) 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3042 ms) 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7464 ms) 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (39483 ms total) 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [----------] Global test environment tear-down 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (39483 ms total) 2024-03-26T05:24:51.680 INFO:tasks.workunit.client.0.smithi136.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-26T05:24:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:51 smithi164 ceph-mon[33554]: pgmap v114: 680 pgs: 160 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T05:24:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:51 smithi164 ceph-mon[33554]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T05:24:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3160365085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[29560]: pgmap v114: 680 pgs: 160 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T05:24:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[29560]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T05:24:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3160365085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[38106]: pgmap v114: 680 pgs: 160 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[38106]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3160365085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:52.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:24:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T05:24:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]: dispatch 2024-03-26T05:24:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]: dispatch 2024-03-26T05:24:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]': finished 2024-03-26T05:24:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T05:24:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T05:24:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]: dispatch 2024-03-26T05:24:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]: dispatch 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]': finished 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi136-74761-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]: dispatch 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]: dispatch 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4"}]': finished 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]: dispatch 2024-03-26T05:24:53.421 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-26T05:24:53.421 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-26T05:24:53.421 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-26T05:24:53.421 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-26T05:24:53.421 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-26T05:24:53.421 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-26T05:24:53.421 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (12086 ms) 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (5374 ms) 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (4349 ms) 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3552 ms) 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3839 ms) 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-26T05:24:53.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (4041 ms) 2024-03-26T05:24:53.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-26T05:24:53.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (8206 ms) 2024-03-26T05:24:53.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (41447 ms total) 2024-03-26T05:24:53.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: 2024-03-26T05:24:53.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-26T05:24:53.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (41447 ms total) 2024-03-26T05:24:53.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout:d4e834fddf2", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro": "centos", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "distro_version": "8", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "frontend_type#0": "beast", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "hostname": "smithi136", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "id": "foo.a", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T05:24:53.590 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "mem_total_kb": "32488364", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "num_handles": "1", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "os": "Linux", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "pid": "2", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "realm_id": "", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "realm_name": "", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "zone_id": "a8944c22-3dfd-4a69-a2c5-4fdbdde5a514", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "zone_name": "default", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "zonegroup_id": "41faacb0-282b-439f-967a-acc7f76dac51", 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "zonegroup_name": "default" 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: }, 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: "task_status": {} 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:53.591 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: } 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: attempt 2 of 20 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (15671 ms) 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (41565 ms total) 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (41565 ms total) 2024-03-26T05:24:53.592 INFO:tasks.workunit.client.0.smithi136.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-26T05:24:54.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: pgmap v117: 744 pgs: 96 unknown, 64 creating+peering, 584 active+clean; 145 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-03-26T05:24:54.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:54.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:54.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]': finished 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]': finished 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2494374994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]: dispatch 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]: dispatch 2024-03-26T05:24:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2278709109' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: pgmap v117: 744 pgs: 96 unknown, 64 creating+peering, 584 active+clean; 145 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]': finished 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]': finished 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2494374994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]: dispatch 2024-03-26T05:24:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2278709109' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: pgmap v117: 744 pgs: 96 unknown, 64 creating+peering, 584 active+clean; 145 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 5 op/s 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-9"}]': finished 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi136-75371-4", "tierpool": "test-rados-api-smithi136-75371-6"}]': finished 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1278274391' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2494374994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]: dispatch 2024-03-26T05:24:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2278709109' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: pgmap v120: 552 pgs: 32 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: 14.2 deep-scrub starts 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: 14.2 deep-scrub ok 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]': finished 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3202526856' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:55.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:55.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:55.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: pgmap v120: 552 pgs: 32 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: 14.2 deep-scrub starts 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: 14.2 deep-scrub ok 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:24:55.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:24:55.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:55.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]': finished 2024-03-26T05:24:55.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T05:24:55.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3202526856' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:55.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:55.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:55.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: pgmap v120: 552 pgs: 32 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.0"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.1"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.2"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.3"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:55.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.4"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.5"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.6"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.7"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.8"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "14.9"}]: dispatch 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: 14.2 deep-scrub starts 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: 14.2 deep-scrub ok 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:24:55.580 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:24:55.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi136-75371-6", "pool2": "test-rados-api-smithi136-75371-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:24:55.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi136-74761-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:55.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-9", "mode": "writeback"}]': finished 2024-03-26T05:24:55.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T05:24:55.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3202526856' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:55.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:55.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:55.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: flushed 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3017 ms) 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: trying... 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930204976 notify_id 412316860420 notifier_gid 25313 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: timed out 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: flushing 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: flushed 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3015 ms) 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-26T05:24:55.970 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: List watches 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: notify2 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930204976 notify_id 425201762314 notifier_gid 25313 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: notify2 done 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: watch_check 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: unwatch2 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: flushing 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: done 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3026 ms) 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: List watches 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: notify2 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: handle_notify cookie 94521930204912 notify_id 438086664195 notifier_gid 25313 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: notify2 done 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: watch_check 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: unwatch2 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: flushing 2024-03-26T05:24:55.971 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: done 2024-03-26T05:24:55.972 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3007 ms) 2024-03-26T05:24:55.972 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18209 ms total) 2024-03-26T05:24:55.972 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: 2024-03-26T05:24:55.972 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-26T05:24:55.972 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (43999 ms total) 2024-03-26T05:24:55.972 INFO:tasks.workunit.client.0.smithi136.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-26T05:24:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: 14.0 deep-scrub starts 2024-03-26T05:24:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: 14.0 deep-scrub ok 2024-03-26T05:24:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: 14.8 deep-scrub starts 2024-03-26T05:24:56.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: 14.8 deep-scrub ok 2024-03-26T05:24:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:24:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T05:24:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: 14.0 deep-scrub starts 2024-03-26T05:24:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: 14.0 deep-scrub ok 2024-03-26T05:24:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: 14.8 deep-scrub starts 2024-03-26T05:24:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: 14.8 deep-scrub ok 2024-03-26T05:24:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:56.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: 14.0 deep-scrub starts 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: 14.0 deep-scrub ok 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: 14.8 deep-scrub starts 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: 14.8 deep-scrub ok 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:56.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:24:56.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:56.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T05:24:56.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]: dispatch 2024-03-26T05:24:57.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:24:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:24:58.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: 14.7 deep-scrub starts 2024-03-26T05:24:58.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: 14.7 deep-scrub ok 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: 14.4 deep-scrub starts 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: 14.4 deep-scrub ok 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: pgmap v122: 644 pgs: 33 unknown, 109 creating+peering, 502 active+clean; 217 MiB data, 926 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: 14.9 deep-scrub starts 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: 14.9 deep-scrub ok 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: 14.6 deep-scrub starts 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: 14.6 deep-scrub ok 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]': finished 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2377897227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1386108084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: 14.7 deep-scrub starts 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: 14.7 deep-scrub ok 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: 14.4 deep-scrub starts 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: 14.4 deep-scrub ok 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: pgmap v122: 644 pgs: 33 unknown, 109 creating+peering, 502 active+clean; 217 MiB data, 926 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: 14.9 deep-scrub starts 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: 14.9 deep-scrub ok 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: 14.6 deep-scrub starts 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: 14.6 deep-scrub ok 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:58.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]': finished 2024-03-26T05:24:58.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T05:24:58.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2377897227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1386108084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] Global test environment set-up. 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: 2024-03-26T05:24:11.925+0000 7fa0b55f5780 -1 failed for service _ceph-mon._tcp 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: 2024-03-26T05:24:11.925+0000 7fa0b55f5780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (70 ms) 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-26T05:24:58.111 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2021 ms) 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2091 ms total) 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: started 0x56022875e600 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: started 0x560228a8a6b0 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: started 2 aios 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: waiting 0x56022875e600 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: waiting 0x560228a8a6b0 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: done. 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (13697 ms) 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (13697 ms total) 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-26T05:24:58.112 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMisc.Exec (171 ms) 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (5 ms) 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (4 ms) 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6298 ms) 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (1 ms) 2024-03-26T05:24:58.113 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-26T05:24:58.114 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (22013 ms) 2024-03-26T05:24:58.114 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28493 ms total) 2024-03-26T05:24:58.114 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: 2024-03-26T05:24:58.114 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [----------] Global test environment tear-down 2024-03-26T05:24:58.114 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (46245 ms total) 2024-03-26T05:24:58.114 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: 14.7 deep-scrub starts 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: 14.7 deep-scrub ok 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: 14.4 deep-scrub starts 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: 14.4 deep-scrub ok 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: pgmap v122: 644 pgs: 33 unknown, 109 creating+peering, 502 active+clean; 217 MiB data, 926 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: 14.9 deep-scrub starts 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: 14.9 deep-scrub ok 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: 14.6 deep-scrub starts 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: 14.6 deep-scrub ok 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-9"}]': finished 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2377897227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1386108084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:24:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:59.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: 14.5 deep-scrub starts 2024-03-26T05:24:59.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: 14.5 deep-scrub ok 2024-03-26T05:24:59.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: pgmap v125: 580 pgs: 96 unknown, 64 creating+peering, 420 active+clean; 217 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 op/s 2024-03-26T05:24:59.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:59.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: 14.3 deep-scrub starts 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: 14.3 deep-scrub ok 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: 14.5 deep-scrub starts 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: 14.5 deep-scrub ok 2024-03-26T05:24:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: pgmap v125: 580 pgs: 96 unknown, 64 creating+peering, 420 active+clean; 217 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 op/s 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: 14.3 deep-scrub starts 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: 14.3 deep-scrub ok 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:24:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: 14.5 deep-scrub starts 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: 14.5 deep-scrub ok 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: pgmap v125: 580 pgs: 96 unknown, 64 creating+peering, 420 active+clean; 217 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 op/s 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi136-74761-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: 14.3 deep-scrub starts 2024-03-26T05:24:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: 14.3 deep-scrub ok 2024-03-26T05:24:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:24:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:24:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:24:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:00.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:00.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:00 smithi164 ceph-mon[33554]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T05:25:00.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:00.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:00.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:00.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:00.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:00.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:00.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[38106]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T05:25:00.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:00.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:00.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:00.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:00.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:00.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[29560]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T05:25:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:01.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: pgmap v128: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:01.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: 14.1 deep-scrub starts 2024-03-26T05:25:01.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: 14.1 deep-scrub ok 2024-03-26T05:25:01.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1979278111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/840925230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:01 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:01.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: pgmap v128: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:01.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: 14.1 deep-scrub starts 2024-03-26T05:25:01.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: 14.1 deep-scrub ok 2024-03-26T05:25:01.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:01.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:01.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1979278111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/840925230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: pgmap v128: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: 14.1 deep-scrub starts 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: 14.1 deep-scrub ok 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:01.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1979278111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/840925230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T05:25:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:01 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:02.122 INFO:tasks.workunit.client.0.smithi136.stdout: misc: Running main() from gmock_main.cc 2024-03-26T05:25:02.122 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-26T05:25:02.122 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [----------] Global test environment set-up. 2024-03-26T05:25:02.122 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-26T05:25:02.122 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-26T05:25:02.122 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.Version (9824 ms) 2024-03-26T05:25:02.122 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-26T05:25:02.122 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (5672 ms) 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.LongName (4857 ms) 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3420 ms) 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3701 ms) 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4589 ms) 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.Exec (3364 ms) 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2868 ms) 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-26T05:25:02.123 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2970 ms) 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3695 ms) 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1993 ms) 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3026 ms) 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [----------] 12 tests from NeoRadosMisc (49979 ms total) 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [----------] Global test environment tear-down 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [==========] 12 tests from 1 test suite ran. (49979 ms total) 2024-03-26T05:25:02.124 INFO:tasks.workunit.client.0.smithi136.stdout: misc: [ PASSED ] 12 tests. 2024-03-26T05:25:02.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:25:02.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:02.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T05:25:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:02 smithi164 ceph-mon[33554]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:02.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[38106]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi136-75062-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi136-74761-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi136-74781-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75106-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:02.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:02 smithi136 ceph-mon[29560]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T05:25:03.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: pgmap v131: 524 pgs: 128 creating+peering, 40 unknown, 356 active+clean; 217 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]': finished 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]': finished 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/362592721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1747292697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi136-74781-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: pgmap v131: 524 pgs: 128 creating+peering, 40 unknown, 356 active+clean; 217 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-26T05:25:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T05:25:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T05:25:03.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]': finished 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]': finished 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]: dispatch 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/362592721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]: dispatch 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1747292697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi136-74781-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: pgmap v131: 524 pgs: 128 creating+peering, 40 unknown, 356 active+clean; 217 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-03-26T05:25:03.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-11"}]': finished 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache", "force_nonempty":""}]': finished 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/362592721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1747292697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi136-74781-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:04.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:04.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:25:04.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]': finished 2024-03-26T05:25:04.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1747292697' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi136-74781-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T05:25:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]: dispatch 2024-03-26T05:25:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]: dispatch 2024-03-26T05:25:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:25:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]': finished 2024-03-26T05:25:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1747292697' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi136-74781-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]: dispatch 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]: dispatch 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-11", "mode": "writeback"}]': finished 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi136-74761-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75106-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1747292697' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi136-74781-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2256797146' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]: dispatch 2024-03-26T05:25:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]: dispatch 2024-03-26T05:25:05.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:05 smithi164 ceph-mon[33554]: pgmap v135: 524 pgs: 32 creating+peering, 136 unknown, 356 active+clean; 217 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-03-26T05:25:05.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:05.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]': finished 2024-03-26T05:25:05.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:05 smithi164 ceph-mon[33554]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T05:25:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:05.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[38106]: pgmap v135: 524 pgs: 32 creating+peering, 136 unknown, 356 active+clean; 217 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-03-26T05:25:05.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:05.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]': finished 2024-03-26T05:25:05.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[38106]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T05:25:05.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:05.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:05.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[29560]: pgmap v135: 524 pgs: 32 creating+peering, 136 unknown, 356 active+clean; 217 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-03-26T05:25:05.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:05.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75106-10", "tierpool":"test-rados-api-smithi136-75106-10-cache"}]': finished 2024-03-26T05:25:05.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[29560]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T05:25:05.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:05.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:06.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: Running main() from gmock_main.cc 2024-03-26T05:25:06.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-26T05:25:06.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [----------] Global test environment set-up. 2024-03-26T05:25:06.422 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolList (11030 ms) 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (6525 ms) 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (4362 ms) 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4337 ms) 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3832 ms) 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5622 ms) 2024-03-26T05:25:06.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5871 ms) 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5658 ms) 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7322 ms) 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [----------] 9 tests from LibRadosPools (54559 ms total) 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [----------] Global test environment tear-down 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (54559 ms total) 2024-03-26T05:25:06.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2347671374' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4139804566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:06.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:25:06.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:06.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T05:25:06.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:06.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2347671374' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4139804566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2347671374' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4139804566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:06.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:07.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:07 smithi164 ceph-mon[33554]: pgmap v138: 492 pgs: 57 creating+peering, 24 unknown, 411 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-26T05:25:07.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:07.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:07 smithi164 ceph-mon[33554]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:07.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:07 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:25:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]': finished 2024-03-26T05:25:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:07 smithi164 ceph-mon[33554]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T05:25:07.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:25:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:25:07.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[29560]: pgmap v138: 492 pgs: 57 creating+peering, 24 unknown, 411 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-26T05:25:07.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:07.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[29560]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:07.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]': finished 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[29560]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[38106]: pgmap v138: 492 pgs: 57 creating+peering, 24 unknown, 411 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[38106]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-11"}]': finished 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi136-74761-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi136-74781-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:07 smithi136 ceph-mon[38106]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T05:25:08.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:08.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:08 smithi164 ceph-mon[33554]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T05:25:08.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:08.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:08 smithi136 ceph-mon[38106]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T05:25:08.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:08.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:08 smithi136 ceph-mon[29560]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: Running main() from gmock_main.cc 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [----------] Global test environment set-up. 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.Limits (9864 ms) 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (6693 ms) 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.ReadOp (3842 ms) 2024-03-26T05:25:09.433 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.SparseRead (3391 ms) 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3702 ms) 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3592 ms) 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (4372 ms) 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2881 ms) 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2970 ms) 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.Trunc (3681 ms) 2024-03-26T05:25:09.434 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.Remove (4027 ms) 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2294 ms) 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.RmXattr (3017 ms) 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3004 ms) 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [----------] 14 tests from NeoRadosIo (57330 ms total) 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [----------] Global test environment tear-down 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [==========] 14 tests from 1 test suite ran. (57330 ms total) 2024-03-26T05:25:09.435 INFO:tasks.workunit.client.0.smithi136.stdout: io: [ PASSED ] 14 tests. 2024-03-26T05:25:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: pgmap v141: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T05:25:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T05:25:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1772374558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/876686224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: pgmap v141: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1772374558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/876686224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: pgmap v141: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T05:25:09.475 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:09.476 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T05:25:09.476 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1772374558' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.476 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.476 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/876686224' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.476 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.476 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:09.476 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:10.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:10.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:10 smithi164 ceph-mon[33554]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T05:25:10.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:10.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[38106]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T05:25:10.489 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:10.489 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.489 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi136-74781-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.489 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi136-74761-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:10.489 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:10 smithi136 ceph-mon[29560]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T05:25:11.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:11 smithi164 ceph-mon[33554]: pgmap v144: 460 pgs: 128 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:11.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:11.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:11 smithi164 ceph-mon[33554]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T05:25:11.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:11.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:11.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[29560]: pgmap v144: 460 pgs: 128 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:11.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:11.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[29560]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T05:25:11.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:11.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[38106]: pgmap v144: 460 pgs: 128 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[38106]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T05:25:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:12.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:25:12.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/380727435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1084018966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:12.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:12.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:12.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T05:25:12.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:12.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/380727435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1084018966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/219796430' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/380727435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]: dispatch 2024-03-26T05:25:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1084018966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:12.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:12.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: pgmap v147: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 KiB/s wr, 55 op/s 2024-03-26T05:25:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:13.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T05:25:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:13.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: pgmap v147: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 KiB/s wr, 55 op/s 2024-03-26T05:25:13.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:13.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:13.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:13.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: pgmap v147: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 KiB/s wr, 55 op/s 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi136-75062-10"}]': finished 2024-03-26T05:25:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi136-74761-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T05:25:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:14.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:14.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:14.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:14 smithi164 ceph-mon[33554]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T05:25:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:14.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:14.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:14.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[29560]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T05:25:14.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:14.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:14.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[38106]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T05:25:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: pgmap v150: 356 pgs: 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T05:25:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:15.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T05:25:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2823838565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/957397012' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: pgmap v150: 356 pgs: 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T05:25:15.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:15.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:15.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:15.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2823838565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/957397012' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: pgmap v150: 356 pgs: 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:15.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T05:25:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2823838565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/957397012' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:16.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]': finished 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]': finished 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]: dispatch 2024-03-26T05:25:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]: dispatch 2024-03-26T05:25:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:16.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]': finished 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]': finished 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]: dispatch 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]: dispatch 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:16.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi136-75062-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-13"}]': finished 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi136-74761-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi136-74781-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]': finished 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]: dispatch 2024-03-26T05:25:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]: dispatch 2024-03-26T05:25:17.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: pgmap v153: 460 pgs: 5 creating+peering, 24 unknown, 431 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 298 KiB/s wr, 43 op/s 2024-03-26T05:25:17.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:25:17.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T05:25:17.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]': finished 2024-03-26T05:25:17.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.0"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.1"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.2"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.3"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.4"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.5"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.6"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.7"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.8"}]: dispatch 2024-03-26T05:25:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.9"}]: dispatch 2024-03-26T05:25:17.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:25:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:25:17.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: pgmap v153: 460 pgs: 5 creating+peering, 24 unknown, 431 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 298 KiB/s wr, 43 op/s 2024-03-26T05:25:17.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]': finished 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.0"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.1"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.2"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.3"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.4"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.5"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.6"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.7"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.8"}]: dispatch 2024-03-26T05:25:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.9"}]: dispatch 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: pgmap v153: 460 pgs: 5 creating+peering, 24 unknown, 431 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 298 KiB/s wr, 43 op/s 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-13", "mode": "writeback"}]': finished 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.0"}]: dispatch 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.1"}]: dispatch 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.2"}]: dispatch 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.3"}]: dispatch 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.4"}]: dispatch 2024-03-26T05:25:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.5"}]: dispatch 2024-03-26T05:25:17.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.6"}]: dispatch 2024-03-26T05:25:17.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.7"}]: dispatch 2024-03-26T05:25:17.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.8"}]: dispatch 2024-03-26T05:25:17.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "156.9"}]: dispatch 2024-03-26T05:25:18.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.0"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.1"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.2"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.3"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.4"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.5"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.6"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.7"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.8"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.9"}]: dispatch 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: 156.5 scrub starts 2024-03-26T05:25:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: 156.5 scrub ok 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3126901889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1637689891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: 156.2 scrub starts 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: 156.2 scrub ok 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: 156.4 scrub starts 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: 156.4 scrub ok 2024-03-26T05:25:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:18 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:18.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.0"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.1"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.2"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.3"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.4"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.5"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.6"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.7"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.8"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.9"}]: dispatch 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: 156.5 scrub starts 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: 156.5 scrub ok 2024-03-26T05:25:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3126901889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1637689891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: 156.2 scrub starts 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: 156.2 scrub ok 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: 156.4 scrub starts 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: 156.4 scrub ok 2024-03-26T05:25:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.0"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.1"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.2"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.3"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.4"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.5"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.6"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.7"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.8"}]: dispatch 2024-03-26T05:25:18.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "156.9"}]: dispatch 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: 156.5 scrub starts 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: 156.5 scrub ok 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3126901889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1637689891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: 156.2 scrub starts 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: 156.2 scrub ok 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: 156.4 scrub starts 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: 156.4 scrub ok 2024-03-26T05:25:18.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:18 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:19.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: pgmap v156: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 331 KiB/s wr, 76 op/s 2024-03-26T05:25:19.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: 156.0 scrub starts 2024-03-26T05:25:19.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: 156.0 scrub ok 2024-03-26T05:25:19.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: 156.8 deep-scrub starts 2024-03-26T05:25:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: 156.8 deep-scrub ok 2024-03-26T05:25:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: 156.3 scrub starts 2024-03-26T05:25:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: 156.3 scrub ok 2024-03-26T05:25:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T05:25:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: pgmap v156: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 331 KiB/s wr, 76 op/s 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: 156.0 scrub starts 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: 156.0 scrub ok 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: 156.8 deep-scrub starts 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: 156.8 deep-scrub ok 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: 156.3 scrub starts 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: 156.3 scrub ok 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: pgmap v156: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 331 KiB/s wr, 76 op/s 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: 156.0 scrub starts 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: 156.0 scrub ok 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: 156.8 deep-scrub starts 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: 156.8 deep-scrub ok 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: 156.3 scrub starts 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: 156.3 scrub ok 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi136-74781-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi136-74761-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T05:25:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:20 smithi164 ceph-mon[33554]: 156.6 scrub starts 2024-03-26T05:25:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:20 smithi164 ceph-mon[33554]: 156.6 scrub ok 2024-03-26T05:25:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:20 smithi164 ceph-mon[33554]: 156.1 scrub starts 2024-03-26T05:25:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:20 smithi164 ceph-mon[33554]: 156.1 scrub ok 2024-03-26T05:25:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:20 smithi164 ceph-mon[33554]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T05:25:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:20.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[29560]: 156.6 scrub starts 2024-03-26T05:25:20.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[29560]: 156.6 scrub ok 2024-03-26T05:25:20.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[29560]: 156.1 scrub starts 2024-03-26T05:25:20.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[29560]: 156.1 scrub ok 2024-03-26T05:25:20.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[29560]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T05:25:20.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[38106]: 156.6 scrub starts 2024-03-26T05:25:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[38106]: 156.6 scrub ok 2024-03-26T05:25:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[38106]: 156.1 scrub starts 2024-03-26T05:25:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[38106]: 156.1 scrub ok 2024-03-26T05:25:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[38106]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T05:25:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:21.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: pgmap v159: 364 pgs: 8 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 29 op/s 2024-03-26T05:25:21.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: 156.9 scrub starts 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: 156.9 scrub ok 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: 156.7 scrub starts 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: 156.7 scrub ok 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3398495796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi136-74761-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1740523430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: pgmap v159: 364 pgs: 8 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 29 op/s 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: 156.9 scrub starts 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: 156.9 scrub ok 2024-03-26T05:25:21.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: 156.7 scrub starts 2024-03-26T05:25:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: 156.7 scrub ok 2024-03-26T05:25:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T05:25:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3398495796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi136-74761-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1740523430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: pgmap v159: 364 pgs: 8 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 29 op/s 2024-03-26T05:25:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: 156.9 scrub starts 2024-03-26T05:25:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: 156.9 scrub ok 2024-03-26T05:25:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: 156.7 scrub starts 2024-03-26T05:25:21.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: 156.7 scrub ok 2024-03-26T05:25:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T05:25:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3398495796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi136-74761-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1740523430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:22.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:25:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:25:22.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3398495796' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi136-74761-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:22.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:22.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[29560]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T05:25:22.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:22.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:22.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3398495796' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi136-74761-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[38106]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T05:25:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:22.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3398495796' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi136-74761-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:22.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi136-74781-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:22 smithi164 ceph-mon[33554]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T05:25:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:23.676 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: pgmap v162: 460 pgs: 96 creating+peering, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-26T05:25:23.676 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:23.677 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T05:25:23.677 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T05:25:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/775025479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2237686262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: pgmap v162: 460 pgs: 96 creating+peering, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-26T05:25:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T05:25:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T05:25:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/775025479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2237686262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: pgmap v162: 460 pgs: 96 creating+peering, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/775025479' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2237686262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: pgmap v165: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-26T05:25:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T05:25:25.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T05:25:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:25 smithi164 ceph-mon[33554]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:25.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: pgmap v165: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-26T05:25:25.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:25.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:25.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T05:25:25.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:25.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[38106]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: pgmap v165: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi136-74781-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi136-74761-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T05:25:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:25.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:25.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T05:25:25.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:25.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:25.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:25 smithi136 ceph-mon[29560]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (5005 ms) 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4662 ms) 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2768 ms) 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (3543 ms) 2024-03-26T05:25:26.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (15978 ms total) 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4575 ms) 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (6392 ms) 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5416 ms) 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (16384 ms total) 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2258 ms) 2024-03-26T05:25:26.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2012 ms) 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1999 ms) 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2019 ms) 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8288 ms total) 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4032 ms) 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3983 ms) 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8015 ms total) 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: 2024-03-26T05:25:26.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-26T05:25:26.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (74585 ms total) 2024-03-26T05:25:26.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-26T05:25:26.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2"}]: dispatch 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2043595596' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1917563886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3738129284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:26.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:26.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T05:25:26.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:26.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2"}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2043595596' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1917563886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3738129284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]: dispatch 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2"}]: dispatch 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2484346419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi136-75062-15"}]': finished 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi136-75086-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2043595596' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1917563886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3738129284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:26.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:27.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:25:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:25:27.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: pgmap v168: 356 pgs: 7 unknown, 349 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: pgmap v168: 356 pgs: 7 unknown, 349 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:25:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:25:27.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: pgmap v168: 356 pgs: 7 unknown, 349 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:27.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:27.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:27.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:27.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74761-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T05:25:27.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi136-74781-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75094-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T05:25:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:25:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T05:25:28.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:28.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T05:25:28.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:28 smithi164 ceph-mon[33554]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T05:25:28.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:25:28.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:25:28.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:28.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T05:25:28.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[38106]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T05:25:28.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:25:28.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:25:28.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T05:25:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[29560]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T05:25:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/140984420' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:25:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T05:25:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:29 smithi164 ceph-mon[33554]: pgmap v171: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]': finished 2024-03-26T05:25:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:29 smithi164 ceph-mon[33554]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T05:25:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/957585466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1628115975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi136-74761-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[38106]: pgmap v171: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]': finished 2024-03-26T05:25:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[38106]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T05:25:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/957585466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1628115975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi136-74761-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[29560]: pgmap v171: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi136-75086-1","app":"app1","key":"key1"}]': finished 2024-03-26T05:25:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[29560]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T05:25:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/957585466' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1628115975' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi136-74761-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:30.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-26T05:25:30.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: seed 75086 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (280 ms) 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (11 ms) 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (12 ms) 2024-03-26T05:25:30.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (14 ms) 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5 ms) 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (4 ms) 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (13 ms) 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (5 ms) 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-03-26T05:25:30.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (13 ms) 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1670 ms) 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (404 ms) 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: done waiting, doing copies 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: done waiting 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60820 ms) 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-03-26T05:25:30.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (4 ms) 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4815 ms) 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-26T05:25:30.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-26T05:25:30.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:30.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1628115975' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi136-74761-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:30.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:30.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:30 smithi164 ceph-mon[33554]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T05:25:30.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:30.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1628115975' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi136-74761-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:30.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:30.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:30 smithi136 ceph-mon[38106]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T05:25:30.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:30.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1628115975' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi136-74761-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:30.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi136-74781-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:30.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:30 smithi136 ceph-mon[29560]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T05:25:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: pgmap v174: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi136-75094-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: pgmap v174: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:31.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T05:25:31.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi136-75094-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: pgmap v174: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75086-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T05:25:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi136-75094-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:32.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:25:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:25:33.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:32 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:32 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:32 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T05:25:33.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:33.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:32 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:33.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:32 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:33.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:32 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T05:25:33.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:32 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:32 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:32 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T05:25:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: pgmap v177: 324 pgs: 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3908442509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1608897386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi136-75094-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: pgmap v177: 324 pgs: 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T05:25:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3908442509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1608897386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi136-75094-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: pgmap v177: 324 pgs: 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3908442509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1608897386' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi136-75094-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75086-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi136-74761-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi136-74781-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:34.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:34.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T05:25:35.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:35 smithi164 ceph-mon[33554]: pgmap v180: 404 pgs: 80 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:35.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]': finished 2024-03-26T05:25:35.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:35 smithi164 ceph-mon[33554]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T05:25:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:35.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[38106]: pgmap v180: 404 pgs: 80 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:35.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]': finished 2024-03-26T05:25:35.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:35.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[38106]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T05:25:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[29560]: pgmap v180: 404 pgs: 80 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T05:25:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pg_num","val":"11"}]': finished 2024-03-26T05:25:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[29560]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T05:25:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:36.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/263242376' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/497771673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/263242376' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T05:25:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T05:25:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/263242376' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/497771673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/263242376' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:36.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/263242376' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/497771673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/263242376' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi136-74761-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75086-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T05:25:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:37.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:25:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:25:37.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-mon[38106]: pgmap v183: 436 pgs: 52 creating+peering, 25 unknown, 359 active+clean; 592 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T05:25:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-mon[38106]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-mon[38106]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T05:25:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-mon[29560]: pgmap v183: 436 pgs: 52 creating+peering, 25 unknown, 359 active+clean; 592 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T05:25:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-mon[29560]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:37 smithi136 ceph-mon[29560]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T05:25:37.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:37 smithi164 ceph-mon[33554]: pgmap v183: 436 pgs: 52 creating+peering, 25 unknown, 359 active+clean; 592 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T05:25:37.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:37 smithi164 ceph-mon[33554]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:37.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:37.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:37 smithi164 ceph-mon[33554]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T05:25:38.445 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP api_snapshots_pp: Running main() from gmock_main.cc 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: seed 75094 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (5006 ms) 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (4662 ms) 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-26T05:25:38.446 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2768 ms) 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (3545 ms) 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4614 ms) 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (20595 ms total) 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (5393 ms) 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4362 ms) 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6169 ms) 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3300 ms) 2024-03-26T05:25:38.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2071 ms) 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1775-g5e70124b/rpm/el9/BUILD/ceph-19.0.0-1775-g5e70124b/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi136-75094-7 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17965 ms) 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (39261 ms total) 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-26T05:25:38.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5071 ms) 2024-03-26T05:25:38.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5080 ms total) 2024-03-26T05:25:38.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: 2024-03-26T05:25:38.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-26T05:25:38.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-26T05:25:38.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3017 ms) 2024-03-26T05:25:38.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-26T05:25:38.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T05:25:38.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T05:25:38.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T05:25:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T05:25:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/181075271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/660695683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T05:25:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T05:25:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T05:25:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T05:25:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/181075271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/660695683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi136-74847-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/181075271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/660695683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:38.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:38.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:39.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: pgmap v186: 340 pgs: 340 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T05:25:39.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:39.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: pgmap v186: 340 pgs: 340 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: pgmap v186: 340 pgs: 340 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi136-74781-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi136-74761-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi136-74847-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1703688586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2595611926' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T05:25:39.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]: dispatch 2024-03-26T05:25:41.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: pgmap v189: 395 pgs: 64 unknown, 331 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:41.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T05:25:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:41.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: pgmap v189: 395 pgs: 64 unknown, 331 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:41.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: pgmap v189: 395 pgs: 64 unknown, 331 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75086-24"}]': finished 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:42.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:25:42.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps snapshots: Running main() from gmock_main.cc 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [----------] Global test environment set-up. 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (15450 ms) 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (6139 ms) 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (5813 ms) 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5638 ms) 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-26T05:25:42.764 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (8206 ms) 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (41246 ms total) 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5666 ms) 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6318 ms) 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7991 ms) 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6038 ms) 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3970 ms) 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-26T05:25:42.765 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi136-76049-11. 2024-03-26T05:25:42.766 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: Waiting for snaps to purge. 2024-03-26T05:25:42.766 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19346 ms) 2024-03-26T05:25:42.766 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (49329 ms total) 2024-03-26T05:25:42.766 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: 2024-03-26T05:25:42.766 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [----------] Global test environment tear-down 2024-03-26T05:25:42.766 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (90575 ms total) 2024-03-26T05:25:42.766 INFO:tasks.workunit.client.0.smithi136.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-26T05:25:42.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:42.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T05:25:42.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1745274117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi136-74781-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2821133159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/767559870' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T05:25:42.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1745274117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi136-74781-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2821133159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/767559870' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:42 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:42.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi136-74847-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:42.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T05:25:42.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1745274117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi136-74781-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2821133159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/767559870' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:42 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: pgmap v192: 396 pgs: 96 unknown, 300 active+clean; 590 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1745274117' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi136-74781-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T05:25:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:44.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: pgmap v192: 396 pgs: 96 unknown, 300 active+clean; 590 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:25:44.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:44.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:44.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1745274117' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi136-74781-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: pgmap v192: 396 pgs: 96 unknown, 300 active+clean; 590 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1745274117' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi136-74781-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi136-74761-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi136-75086-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T05:25:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi136-74847-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T05:25:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:44.471 INFO:tasks.workunit.client.0.smithi136.stdout: expected=1 2024-03-26T05:25:44.471 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:f905c69b:::2:head 2024-03-26T05:25:44.471 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:f905c69b:::2:head expected=12:f905c69b:::2:head 2024-03-26T05:25:44.471 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:f905c69b:::2:head -> 2 2024-03-26T05:25:44.471 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=2 expected=2 2024-03-26T05:25:44.471 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:b29083e3:::5:head 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:b29083e3:::5:head expected=12:b29083e3:::5:head 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:b29083e3:::5:head -> 5 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=5 expected=5 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:89d3ae78:::11:head 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:89d3ae78:::11:head expected=12:89d3ae78:::11:head 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:89d3ae78:::11:head -> 11 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=11 expected=11 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:863748b0:::15:head 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:863748b0:::15:head expected=12:863748b0:::15:head 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:863748b0:::15:head -> 15 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=15 expected=15 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:6cac518f:::0:head 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:6cac518f:::0:head expected=12:6cac518f:::0:head 2024-03-26T05:25:44.472 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:6cac518f:::0:head -> 0 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=0 expected=0 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:62a1935d:::14:head 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:62a1935d:::14:head expected=12:62a1935d:::14:head 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:62a1935d:::14:head -> 14 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=14 expected=14 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:5c6b0b28:::7:head 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:5c6b0b28:::7:head expected=12:5c6b0b28:::7:head 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:5c6b0b28:::7:head -> 7 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=7 expected=7 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : seek to 12:52ea6a34:::10:head 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : cursor()=12:52ea6a34:::10:head expected=12:52ea6a34:::10:head 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: > 12:52ea6a34:::10:head -> 10 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: : entry=10 expected=10 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (231 ms) 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-26T05:25:44.473 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (71412 ms) 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: split 0/5 -> MIN 12:33333333::::head 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: split 1/5 -> 12:33333333::::head 12:66666666::::head 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: split 2/5 -> 12:66666666::::head 12:99999999::::head 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: split 3/5 -> 12:99999999::::head 12:cccccccc::::head 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: split 4/5 -> 12:cccccccc::::head MAX 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (6008 ms) 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [----------] 7 tests from LibRadosList (79190 ms total) 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1033 ms) 2024-03-26T05:25:44.474 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo1 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo2 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo3 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo4 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo5 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo1 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: myset foo6,foo7 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo7 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: foo6 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns1:foo4 2024-03-26T05:25:44.475 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns1:foo5 2024-03-26T05:25:44.476 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns2:foo7 2024-03-26T05:25:44.476 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns2:foo6 2024-03-26T05:25:44.476 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: ns1:foo1 2024-03-26T05:25:44.476 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: :foo1 2024-03-26T05:25:44.476 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: :foo2 2024-03-26T05:25:44.476 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: :foo3 2024-03-26T05:25:44.476 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (30 ms) 2024-03-26T05:25:45.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:45 smithi164 ceph-mon[33554]: pgmap v195: 268 pgs: 8 unknown, 260 active+clean; 590 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:45.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:45.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:45 smithi164 ceph-mon[33554]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T05:25:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2436759940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi136-74761-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2447869341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi136-74781-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/600306912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[38106]: pgmap v195: 268 pgs: 8 unknown, 260 active+clean; 590 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:45.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:45.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[38106]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T05:25:45.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2436759940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi136-74761-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2447869341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi136-74781-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/600306912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[29560]: pgmap v195: 268 pgs: 8 unknown, 260 active+clean; 590 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[29560]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]: dispatch 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2436759940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi136-74761-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2447869341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi136-74781-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/600306912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:45.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2436759940' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi136-74761-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2447869341' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi136-74781-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2436759940' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi136-74761-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2447869341' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi136-74781-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T05:25:46.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2709468613' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi136-75094-16"}]': finished 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2436759940' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi136-74761-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2447869341' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi136-74781-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi136-75086-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T05:25:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:25:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:25:48.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: pgmap v198: 356 pgs: 22 unknown, 334 active+clean; 598 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2 op/s 2024-03-26T05:25:48.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:48.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: pgmap v198: 356 pgs: 22 unknown, 334 active+clean; 598 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2 op/s 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: pgmap v198: 356 pgs: 22 unknown, 334 active+clean; 598 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2 op/s 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1938166452' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]: dispatch 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:48.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:48.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:49.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4088492338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3445632061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: pgmap v201: 324 pgs: 64 unknown, 260 active+clean; 590 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:25:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:25:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:25:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T05:25:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3995185894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:49.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:49.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:25:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T05:25:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4088492338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3445632061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: pgmap v201: 324 pgs: 64 unknown, 260 active+clean; 590 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3995185894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi136-74847-2"}]': finished 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi136-74761-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4088492338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3445632061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: pgmap v201: 324 pgs: 64 unknown, 260 active+clean; 590 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:25:49.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi136-75086-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi136-74781-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T05:25:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3995185894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:49.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:50.464 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-26T05:25:50.464 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 1 0 2024-03-26T05:25:50.464 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 10 0 2024-03-26T05:25:50.464 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 13 0 2024-03-26T05:25:50.464 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 7 0 2024-03-26T05:25:50.464 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 14 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 0 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 15 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 11 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 5 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 8 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 6 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 3 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 4 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 12 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 9 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 2 0 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (42 ms) 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [----------] 3 tests from LibRadosListEC (1105 ms total) 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-26T05:25:50.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-26T05:25:50.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3014 ms) 2024-03-26T05:25:50.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [----------] 1 test from LibRadosListNP (3014 ms total) 2024-03-26T05:25:50.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: 2024-03-26T05:25:50.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [----------] Global test environment tear-down 2024-03-26T05:25:50.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [==========] 11 tests from 3 test suites ran. (98657 ms total) 2024-03-26T05:25:50.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_list: [ PASSED ] 11 tests. 2024-03-26T05:25:50.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:50.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3995185894' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:50.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:50.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:50.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T05:25:50.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3995185894' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi136-74761-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-74847-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:50.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T05:25:50.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3995185894' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T05:25:50.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T05:25:50.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:50.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:50.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: pgmap v204: 308 pgs: 48 unknown, 260 active+clean; 590 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T05:25:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T05:25:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4144271212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: pgmap v204: 308 pgs: 48 unknown, 260 active+clean; 590 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4144271212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: pgmap v204: 308 pgs: 48 unknown, 260 active+clean; 590 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi136-74847-3","pool2":"test-rados-api-smithi136-74847-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T05:25:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T05:25:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4144271212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:25:52.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]': finished 2024-03-26T05:25:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:25:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]': finished 2024-03-26T05:25:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T05:25:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi136-74781-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:53.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:25:53.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]: dispatch 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi136-75086-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-13"}]': finished 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/594101342' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]: dispatch 2024-03-26T05:25:53.425 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: Running main() from gmock_main.cc 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [----------] Global test environment set-up. 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.TooBig (9914 ms) 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (7749 ms) 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (4388 ms) 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-26T05:25:53.426 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (4306 ms) 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3833 ms) 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3256 ms) 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (4232 ms) 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3179 ms) 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3331 ms) 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3146 ms) 2024-03-26T05:25:53.427 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3039 ms) 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2286 ms) 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3016 ms) 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3005 ms) 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.Flush (2974 ms) 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3022 ms) 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-26T05:25:53.428 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3010 ms) 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3044 ms) 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2940 ms) 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2999 ms) 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3038 ms) 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2961 ms) 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-26T05:25:53.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3027 ms) 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3008 ms) 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2956 ms) 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3034 ms) 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [----------] 26 tests from LibRadosAio (94693 ms total) 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-26T05:25:53.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-26T05:25:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: pgmap v207: 300 pgs: 32 creating+peering, 268 active+clean; 599 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 21 op/s 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4229972673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi136-74781-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: pgmap v207: 300 pgs: 32 creating+peering, 268 active+clean; 599 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 21 op/s 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4229972673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi136-74781-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: pgmap v207: 300 pgs: 32 creating+peering, 268 active+clean; 599 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 21 op/s 2024-03-26T05:25:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:25:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi136-74761-27"}]': finished 2024-03-26T05:25:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T05:25:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4229972673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi136-74781-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:54.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:54.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:54.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:25:55.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: pgmap v210: 276 pgs: 40 unknown, 236 active+clean; 599 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4229972673' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi136-74781-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:56.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: pgmap v210: 276 pgs: 40 unknown, 236 active+clean; 599 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T05:25:56.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4229972673' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi136-74781-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: pgmap v210: 276 pgs: 40 unknown, 236 active+clean; 599 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4229972673' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi136-74781-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi136-74761-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:56.429 INFO:tasks.workunit.client.0.smithi136.stdout: P (68086 ms total) 2024-03-26T05:25:56.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: 2024-03-26T05:25:56.429 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (118 ms) 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (118 ms total) 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (258 ms) 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (5 ms) 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (263 ms total) 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-26T05:25:56.430 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (42 ms) 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (2 ms) 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (44 ms total) 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (45 ms) 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (3 ms) 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (48 ms total) 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-26T05:25:56.431 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-26T05:25:56.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (991 ms) 2024-03-26T05:25:56.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (991 ms total) 2024-03-26T05:25:56.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: 2024-03-26T05:25:56.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-26T05:25:56.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (104558 ms total) 2024-03-26T05:25:56.432 INFO:tasks.workunit.client.0.smithi136.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-26T05:25:56.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:56.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2505663328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:57.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2505663328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1499659686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]: dispatch 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi136-74761-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi136-75086-36"}]': finished 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2505663328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:25:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:57.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:25:57.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:25:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:25:57.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:57 smithi164 ceph-mon[33554]: pgmap v213: 268 pgs: 7 unknown, 261 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:57.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:57 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:57.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:57.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:57 smithi164 ceph-mon[33554]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T05:25:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:25:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:25:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[29560]: pgmap v213: 268 pgs: 7 unknown, 261 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[29560]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[38106]: pgmap v213: 268 pgs: 7 unknown, 261 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi136-74781-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[38106]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T05:25:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:25:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:25:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:59 smithi164 ceph-mon[33554]: pgmap v216: 308 pgs: 40 unknown, 268 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-03-26T05:25:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]': finished 2024-03-26T05:25:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:59 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]: dispatch 2024-03-26T05:25:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:59 smithi164 ceph-mon[33554]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T05:25:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]: dispatch 2024-03-26T05:25:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:59 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:25:59 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:59.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[29560]: pgmap v216: 308 pgs: 40 unknown, 268 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]': finished 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]: dispatch 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[29560]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]: dispatch 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[38106]: pgmap v216: 308 pgs: 40 unknown, 268 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-15"}]': finished 2024-03-26T05:25:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]: dispatch 2024-03-26T05:25:59.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[38106]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T05:25:59.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]: dispatch 2024-03-26T05:25:59.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:59.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:25:59.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:25:59 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:00.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]': finished 2024-03-26T05:26:00.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:26:00.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T05:26:00.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:26:00.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:26:00.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/956889418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi136-74781-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:00.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:26:00.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/956889418' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi136-74781-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:00.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:00 smithi164 ceph-mon[33554]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T05:26:00.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:00.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]': finished 2024-03-26T05:26:00.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:26:00.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T05:26:00.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:26:00.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/956889418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi136-74781-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/956889418' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi136-74781-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[38106]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-15", "mode": "writeback"}]': finished 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1276200380' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]: dispatch 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/956889418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi136-74781-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi136-74761-28"}]': finished 2024-03-26T05:26:00.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/956889418' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi136-74781-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:00.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:00 smithi136 ceph-mon[29560]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T05:26:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: pgmap v219: 300 pgs: 32 unknown, 268 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:01.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: pgmap v219: 300 pgs: 32 unknown, 268 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: pgmap v219: 300 pgs: 32 unknown, 268 active+clean; 607 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi136-74761-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T05:26:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:26:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]: dispatch 2024-03-26T05:26:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:02.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:26:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:26:02.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:02 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]': finished 2024-03-26T05:26:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:26:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:02 smithi164 ceph-mon[33554]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T05:26:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2537569909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi136-74781-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:03.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:03.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]': finished 2024-03-26T05:26:03.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:26:03.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[38106]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2537569909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi136-74781-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-15"}]': finished 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1491186151' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[29560]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]: dispatch 2024-03-26T05:26:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2537569909' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi136-74781-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout:hotsECPP.SnapRemovePP (2023 ms) 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2002 ms) 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2315 ms) 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9357 ms total) 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4037 ms) 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-26T05:26:03.423 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3936 ms) 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4015 ms) 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11988 ms total) 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (111533 ms total) 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-26T05:26:03.424 INFO:tasks.workunit.client.0.smithi136.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T05:26:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:03 smithi164 ceph-mon[33554]: pgmap v222: 260 pgs: 260 active+clean; 583 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:26:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2537569909' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi136-74781-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:03 smithi164 ceph-mon[33554]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T05:26:04.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[38106]: pgmap v222: 260 pgs: 260 active+clean; 583 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:04.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:04.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:04.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:26:04.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2537569909' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi136-74781-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:04.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[38106]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T05:26:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[29560]: pgmap v222: 260 pgs: 260 active+clean; 583 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi136-74761-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi136-75094-21"}]': finished 2024-03-26T05:26:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2537569909' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi136-74781-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:03 smithi136 ceph-mon[29560]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T05:26:05.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:05 smithi164 ceph-mon[33554]: pgmap v225: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:05.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:05 smithi164 ceph-mon[33554]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T05:26:05.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:05 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[38106]: pgmap v225: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[38106]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T05:26:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[29560]: pgmap v225: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[29560]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T05:26:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:05 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:06.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:06.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3808804451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:06.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T05:26:06.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3808804451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:06.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3808804451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi136-74781-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1786657058' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:06.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]: dispatch 2024-03-26T05:26:07.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:26:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:26:07.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[38106]: pgmap v228: 292 pgs: 27 creating+peering, 9 unknown, 256 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]': finished 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[38106]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]: dispatch 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]: dispatch 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[29560]: pgmap v228: 292 pgs: 27 creating+peering, 9 unknown, 256 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]': finished 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[29560]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]: dispatch 2024-03-26T05:26:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]: dispatch 2024-03-26T05:26:07.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:07 smithi164 ceph-mon[33554]: pgmap v228: 292 pgs: 27 creating+peering, 9 unknown, 256 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:07.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-17"}]': finished 2024-03-26T05:26:07.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi136-74761-29"}]': finished 2024-03-26T05:26:07.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:07 smithi164 ceph-mon[33554]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T05:26:07.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]: dispatch 2024-03-26T05:26:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]: dispatch 2024-03-26T05:26:08.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]': finished 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2337603387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi136-74781-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]': finished 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2337603387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi136-74781-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-17", "mode": "writeback"}]': finished 2024-03-26T05:26:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi136-74761-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T05:26:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2337603387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi136-74781-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:09.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:09 smithi164 ceph-mon[33554]: pgmap v231: 260 pgs: 260 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:09.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:09.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:09.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2337603387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi136-74781-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:10.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:09 smithi136 ceph-mon[38106]: pgmap v231: 260 pgs: 260 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:10.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:10.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2337603387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi136-74781-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:09 smithi136 ceph-mon[29560]: pgmap v231: 260 pgs: 260 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2337603387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi136-74781-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:10 smithi164 ceph-mon[33554]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T05:26:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:10 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]': finished 2024-03-26T05:26:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:10 smithi164 ceph-mon[33554]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T05:26:11.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:11.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:11.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[29560]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]': finished 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[29560]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[38106]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]: dispatch 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi136-74761-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:11.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-17"}]': finished 2024-03-26T05:26:11.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:10 smithi136 ceph-mon[38106]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T05:26:11.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:11 smithi164 ceph-mon[33554]: pgmap v234: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:11.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:11 smithi164 ceph-mon[33554]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T05:26:11.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3681056263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:12.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:11 smithi136 ceph-mon[38106]: pgmap v234: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:12.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:11 smithi136 ceph-mon[38106]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T05:26:12.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3681056263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:12.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:11 smithi136 ceph-mon[29560]: pgmap v234: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:11 smithi136 ceph-mon[29560]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T05:26:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3681056263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:12.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:26:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:26:12.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:12 smithi164 ceph-mon[33554]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T05:26:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:13.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:13.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:13.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[38106]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi136-74781-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[29560]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:13.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-26T05:26:13.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-26T05:26:13.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-26T05:26:13.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-26T05:26:13.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-26T05:26:13.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (9910 ms) 2024-03-26T05:26:13.447 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (34295 ms) 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6164 ms) 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2285 ms) 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3008 ms) 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3017 ms) 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2978 ms) 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3016 ms) 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3009 ms) 2024-03-26T05:26:13.448 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3045 ms) 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2936 ms) 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3004 ms) 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3040 ms) 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2963 ms) 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3016 ms) 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3011 ms) 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2964 ms) 2024-03-26T05:26:13.449 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3031 ms) 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3006 ms) 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3010 ms) 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3207 ms) 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2756 ms) 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-26T05:26:13.450 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3010 ms) 2024-03-26T05:26:13.451 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-26T05:26:13.451 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2989 ms) 2024-03-26T05:26:13.451 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-26T05:26:13.451 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3004 ms) 2024-03-26T05:26:13.451 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-26T05:26:13.451 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3008 ms) 2024-03-26T05:26:13.451 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-26T05:26:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: pgmap v237: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 583 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: pgmap v237: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 583 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T05:26:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:14.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: pgmap v237: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 583 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:14.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T05:26:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2435902348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]: dispatch 2024-03-26T05:26:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:15.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: pgmap v240: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3055283956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:15.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: pgmap v240: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3055283956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: pgmap v240: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi136-74761-30"}]': finished 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3055283956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:15.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:15.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:16.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:16.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]': finished 2024-03-26T05:26:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T05:26:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]: dispatch 2024-03-26T05:26:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]: dispatch 2024-03-26T05:26:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]': finished 2024-03-26T05:26:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]': finished 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]: dispatch 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]: dispatch 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]': finished 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi136-74781-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-19"}]': finished 2024-03-26T05:26:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi136-74761-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T05:26:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]: dispatch 2024-03-26T05:26:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]: dispatch 2024-03-26T05:26:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-19", "mode": "writeback"}]': finished 2024-03-26T05:26:17.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:26:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:26:17.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: pgmap v243: 292 pgs: 11 unknown, 281 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T05:26:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T05:26:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/751944494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: pgmap v243: 292 pgs: 11 unknown, 281 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/751944494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: pgmap v243: 292 pgs: 11 unknown, 281 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi136-74761-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]: dispatch 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/751944494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:18.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:18 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:18.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:18 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:18.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]': finished 2024-03-26T05:26:18.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:18.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:18 smithi164 ceph-mon[33554]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T05:26:19.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:19.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:19.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]': finished 2024-03-26T05:26:19.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:19.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[38106]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T05:26:19.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:19.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:19.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-19"}]': finished 2024-03-26T05:26:19.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi136-74781-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:19.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:18 smithi136 ceph-mon[29560]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T05:26:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:19 smithi164 ceph-mon[33554]: pgmap v246: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:19 smithi164 ceph-mon[33554]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T05:26:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:19 smithi136 ceph-mon[29560]: pgmap v246: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:19 smithi136 ceph-mon[29560]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T05:26:20.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:20.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:19 smithi136 ceph-mon[38106]: pgmap v246: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:19 smithi136 ceph-mon[38106]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T05:26:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:21.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: pgmap v249: 228 pgs: 228 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T05:26:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3231316526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: pgmap v249: 228 pgs: 228 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3231316526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: pgmap v249: 228 pgs: 228 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:21.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T05:26:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3350182892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3231316526' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]: dispatch 2024-03-26T05:26:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:22.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:26:22.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:22.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:22.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:22.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:22.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi136-74761-31"}]': finished 2024-03-26T05:26:22.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi136-74781-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: pgmap v252: 292 pgs: 64 creating+peering, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: pgmap v252: 292 pgs: 64 creating+peering, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T05:26:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:24.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: pgmap v252: 292 pgs: 64 creating+peering, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi136-74761-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T05:26:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:25.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]': finished 2024-03-26T05:26:25.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]: dispatch 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]: dispatch 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi136-74781-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]': finished 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi136-74781-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]': finished 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]: dispatch 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]: dispatch 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi136-74781-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]': finished 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi136-74781-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:25.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-21"}]': finished 2024-03-26T05:26:25.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]: dispatch 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]: dispatch 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi136-74781-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi136-74761-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-21", "mode": "writeback"}]': finished 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi136-74781-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:26.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: pgmap v255: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd=[{ 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: "prefix": "osd pool set", 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: "pool": "PoolEIOFlag_smithi136-74781-33", 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: "var": "eio", 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: "val": "true" 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: }]: dispatch 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd='[{ 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: "prefix": "osd pool set", 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: "pool": "PoolEIOFlag_smithi136-74781-33", 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: "var": "eio", 2024-03-26T05:26:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: "val": "true" 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: }]': finished 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: pgmap v255: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd=[{ 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: "prefix": "osd pool set", 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: "pool": "PoolEIOFlag_smithi136-74781-33", 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: "var": "eio", 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: "val": "true" 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: }]: dispatch 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:26.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd='[{ 2024-03-26T05:26:26.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: "prefix": "osd pool set", 2024-03-26T05:26:26.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: "pool": "PoolEIOFlag_smithi136-74781-33", 2024-03-26T05:26:26.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: "var": "eio", 2024-03-26T05:26:26.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: "val": "true" 2024-03-26T05:26:26.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: }]': finished 2024-03-26T05:26:26.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T05:26:26.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:26.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: pgmap v255: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd=[{ 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: "prefix": "osd pool set", 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: "pool": "PoolEIOFlag_smithi136-74781-33", 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: "var": "eio", 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: "val": "true" 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: }]: dispatch 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3592900181' entity='client.admin' cmd='[{ 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: "prefix": "osd pool set", 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: "pool": "PoolEIOFlag_smithi136-74781-33", 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: "var": "eio", 2024-03-26T05:26:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: "val": "true" 2024-03-26T05:26:26.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: }]': finished 2024-03-26T05:26:26.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T05:26:26.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:26.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]: dispatch 2024-03-26T05:26:27.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]': finished 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[29560]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]': finished 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[38106]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:26 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-21"}]': finished 2024-03-26T05:26:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:26 smithi164 ceph-mon[33554]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T05:26:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:27.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:26:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:26:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[38106]: pgmap v258: 300 pgs: 2 unknown, 2 creating+activating, 4 creating+peering, 292 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-03-26T05:26:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[38106]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/159134476' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[29560]: pgmap v258: 300 pgs: 2 unknown, 2 creating+activating, 4 creating+peering, 292 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[29560]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/159134476' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:27 smithi164 ceph-mon[33554]: pgmap v258: 300 pgs: 2 unknown, 2 creating+activating, 4 creating+peering, 292 active+clean; 584 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-03-26T05:26:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:27 smithi164 ceph-mon[33554]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T05:26:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3595011191' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]: dispatch 2024-03-26T05:26:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/159134476' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: pgmap v261: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:26:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi136-74761-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: pgmap v261: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T05:26:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi136-74761-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:29.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: pgmap v261: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:26:29.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi136-74761-32"}]': finished 2024-03-26T05:26:29.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi136-74781-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T05:26:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi136-74761-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:31.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:31.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi136-74761-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:30 smithi136 ceph-mon[38106]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T05:26:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi136-74761-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:31.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:31.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi136-74761-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:31.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:30 smithi136 ceph-mon[29560]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T05:26:31.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi136-74761-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:31.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:31.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi136-74761-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:31.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:30 smithi164 ceph-mon[33554]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T05:26:31.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi136-74761-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:32.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: pgmap v264: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:32.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/145925615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi136-74761-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: pgmap v264: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/145925615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi136-74761-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: pgmap v264: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/145925615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi136-74761-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi136-74781-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T05:26:32.077 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:32.078 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:32.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:26:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:26:33.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[38106]: pgmap v267: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:26:33.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:33.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]': finished 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[38106]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]: dispatch 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]: dispatch 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[29560]: pgmap v267: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]': finished 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[29560]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]: dispatch 2024-03-26T05:26:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]: dispatch 2024-03-26T05:26:33.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:33 smithi164 ceph-mon[33554]: pgmap v267: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:26:33.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:33.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-23"}]': finished 2024-03-26T05:26:33.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:33 smithi164 ceph-mon[33554]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T05:26:33.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]: dispatch 2024-03-26T05:26:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]: dispatch 2024-03-26T05:26:34.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:34.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]': finished 2024-03-26T05:26:34.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[38106]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T05:26:34.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:34.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2315753500' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:34.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]': finished 2024-03-26T05:26:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[29560]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T05:26:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2315753500' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:34.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:34 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:34.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-23", "mode": "writeback"}]': finished 2024-03-26T05:26:34.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:34 smithi164 ceph-mon[33554]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T05:26:34.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2315753500' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:35.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[38106]: pgmap v270: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[38106]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[29560]: pgmap v270: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[29560]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T05:26:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:35.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:35.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:35.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:35 smithi164 ceph-mon[33554]: pgmap v270: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:26:35.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:35.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:35 smithi164 ceph-mon[33554]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T05:26:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]: dispatch 2024-03-26T05:26:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3828259521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi136-74761-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]': finished 2024-03-26T05:26:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi136-74761-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi136-74761-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3828259521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:36.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi136-74761-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:36.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:36.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:36.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]': finished 2024-03-26T05:26:36.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi136-74761-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:36.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T05:26:36.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi136-74761-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:36.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3893791476' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi136-74761-33"}]': finished 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3828259521' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]: dispatch 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi136-74761-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-23"}]': finished 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi136-74761-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T05:26:36.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi136-74761-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:37.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:26:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:26:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:26:38.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:37 smithi136 ceph-mon[38106]: pgmap v273: 324 pgs: 30 creating+peering, 5 unknown, 289 active+clean; 584 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T05:26:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:37 smithi136 ceph-mon[29560]: pgmap v273: 324 pgs: 30 creating+peering, 5 unknown, 289 active+clean; 584 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T05:26:38.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:37 smithi164 ceph-mon[33554]: pgmap v273: 324 pgs: 30 creating+peering, 5 unknown, 289 active+clean; 584 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T05:26:39.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[38106]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T05:26:39.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/981764650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:39.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:39.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi136-74761-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:39.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/981764650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:39.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[38106]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T05:26:39.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:39.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:39.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[29560]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T05:26:39.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/981764650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:39.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:39.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi136-74761-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:39.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/981764650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:39.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[29560]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T05:26:39.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:39.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:39.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:38 smithi164 ceph-mon[33554]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T05:26:39.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/981764650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:39.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:38 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:39.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi136-74761-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:39.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/981764650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi136-74781-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:39.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:38 smithi164 ceph-mon[33554]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T05:26:39.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:39.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:40.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:39 smithi136 ceph-mon[38106]: pgmap v276: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:40.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:39 smithi136 ceph-mon[38106]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T05:26:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:39 smithi136 ceph-mon[29560]: pgmap v276: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:39 smithi136 ceph-mon[29560]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T05:26:40.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:39 smithi164 ceph-mon[33554]: pgmap v276: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:40.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:39 smithi164 ceph-mon[33554]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T05:26:41.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[29560]: pgmap v279: 332 pgs: 40 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:41.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[29560]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T05:26:41.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[38106]: pgmap v279: 332 pgs: 40 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[38106]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T05:26:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:41.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:41 smithi164 ceph-mon[33554]: pgmap v279: 332 pgs: 40 unknown, 32 creating+peering, 260 active+clean; 584 KiB data, 540 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:41.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:41 smithi164 ceph-mon[33554]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T05:26:41.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:42.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:26:42.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]': finished 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]: dispatch 2024-03-26T05:26:42.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]': finished 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T05:26:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]: dispatch 2024-03-26T05:26:42.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]: dispatch 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1415914101' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi136-74761-34"}]': finished 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-25"}]': finished 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T05:26:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]: dispatch 2024-03-26T05:26:43.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: pgmap v282: 260 pgs: 260 active+clean; 583 KiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]: dispatch 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/172627524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi136-74761-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]': finished 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi136-74761-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:43.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T05:26:43.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi136-74761-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:43.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:43.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: pgmap v282: 260 pgs: 260 active+clean; 583 KiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]: dispatch 2024-03-26T05:26:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/172627524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi136-74761-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]': finished 2024-03-26T05:26:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:44.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi136-74761-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:44.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T05:26:44.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi136-74761-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:44.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:44.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:44.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: pgmap v282: 260 pgs: 260 active+clean; 583 KiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:44.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:44.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]: dispatch 2024-03-26T05:26:44.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/172627524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:44.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:44.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:44.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:44.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi136-74761-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:44.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:44.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:44.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-25", "mode": "writeback"}]': finished 2024-03-26T05:26:44.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:44.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi136-74761-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:44.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T05:26:44.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi136-74761-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:44.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:44.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:45.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:45 smithi164 ceph-mon[33554]: pgmap v285: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:45 smithi164 ceph-mon[33554]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T05:26:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1684318330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:45.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[38106]: pgmap v285: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:45.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:45.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[38106]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1684318330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[29560]: pgmap v285: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[29560]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]: dispatch 2024-03-26T05:26:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1684318330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:46.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:46 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi136-74761-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]': finished 2024-03-26T05:26:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1684318330' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:46 smithi164 ceph-mon[33554]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T05:26:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi136-74761-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]': finished 2024-03-26T05:26:46.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1684318330' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[38106]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T05:26:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi136-74761-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-25"}]': finished 2024-03-26T05:26:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1684318330' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:46 smithi136 ceph-mon[29560]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T05:26:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:26:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:26:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:26:47.574 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:47 smithi136 ceph-mon[29560]: pgmap v288: 332 pgs: 6 creating+peering, 8 unknown, 318 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:47 smithi136 ceph-mon[29560]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T05:26:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1120070919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:47 smithi136 ceph-mon[38106]: pgmap v288: 332 pgs: 6 creating+peering, 8 unknown, 318 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:47 smithi136 ceph-mon[38106]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T05:26:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1120070919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:47.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:47 smithi164 ceph-mon[33554]: pgmap v288: 332 pgs: 6 creating+peering, 8 unknown, 318 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:47 smithi164 ceph-mon[33554]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T05:26:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1120070919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1120070919' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T05:26:48.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/485727542' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1120070919' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:48.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/485727542' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1120070919' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:48.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T05:26:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:26:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T05:26:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]: dispatch 2024-03-26T05:26:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/485727542' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:50.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: pgmap v291: 356 pgs: 64 unknown, 292 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/485727542' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: pgmap v291: 356 pgs: 64 unknown, 292 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/485727542' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:50.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: pgmap v291: 356 pgs: 64 unknown, 292 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3922447719' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi136-74761-35"}]': finished 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/485727542' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:50.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:50.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: pgmap v294: 388 pgs: 96 unknown, 292 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:51.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T05:26:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/354747236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: pgmap v294: 388 pgs: 96 unknown, 292 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/354747236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: pgmap v294: 388 pgs: 96 unknown, 292 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi136-74761-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/354747236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:52.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:26:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]': finished 2024-03-26T05:26:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/354747236' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:52.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]: dispatch 2024-03-26T05:26:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T05:26:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]: dispatch 2024-03-26T05:26:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]': finished 2024-03-26T05:26:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:52 smithi164 ceph-mon[33554]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T05:26:53.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]': finished 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/354747236' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]: dispatch 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]: dispatch 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]': finished 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[38106]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-27"}]': finished 2024-03-26T05:26:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/354747236' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi136-74781-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]: dispatch 2024-03-26T05:26:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T05:26:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]: dispatch 2024-03-26T05:26:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:26:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi136-74761-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-27", "mode": "writeback"}]': finished 2024-03-26T05:26:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:52 smithi136 ceph-mon[29560]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T05:26:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:53 smithi164 ceph-mon[33554]: pgmap v297: 420 pgs: 32 creating+peering, 388 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T05:26:53.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:53 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:53 smithi164 ceph-mon[33554]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T05:26:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[38106]: pgmap v297: 420 pgs: 32 creating+peering, 388 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T05:26:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[38106]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[29560]: pgmap v297: 420 pgs: 32 creating+peering, 388 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[29560]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:26:55.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:55 smithi164 ceph-mon[33554]: pgmap v300: 364 pgs: 8 unknown, 356 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T05:26:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:55 smithi164 ceph-mon[33554]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T05:26:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:55.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[38106]: pgmap v300: 364 pgs: 8 unknown, 356 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T05:26:55.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:55.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[38106]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T05:26:55.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[29560]: pgmap v300: 364 pgs: 8 unknown, 356 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[29560]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]: dispatch 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:57.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:56 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:57.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]': finished 2024-03-26T05:26:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:56 smithi164 ceph-mon[33554]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T05:26:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:56 smithi164 ceph-mon[33554]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]': finished 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[38106]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[38106]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:26:57.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-27"}]': finished 2024-03-26T05:26:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[29560]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T05:26:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2875918136' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]: dispatch 2024-03-26T05:26:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi136-74761-36"}]': finished 2024-03-26T05:26:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:56 smithi136 ceph-mon[29560]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T05:26:57.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:26:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:26:58.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: pgmap v303: 292 pgs: 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2055372017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: pgmap v303: 292 pgs: 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2055372017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: pgmap v303: 292 pgs: 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:58.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2055372017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:58.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:26:59.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:58 smithi164 ceph-mon[33554]: pgmap v306: 292 pgs: 64 unknown, 228 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:58 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:26:58 smithi164 ceph-mon[33554]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T05:26:59.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[38106]: pgmap v306: 292 pgs: 64 unknown, 228 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[38106]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[29560]: pgmap v306: 292 pgs: 64 unknown, 228 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi136-74781-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:26:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:26:58 smithi136 ceph-mon[29560]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T05:27:00.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:00 smithi136 ceph-mon[38106]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T05:27:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:00 smithi136 ceph-mon[29560]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T05:27:00.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi136-74761-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:00.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:00 smithi164 ceph-mon[33554]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T05:27:01.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[38106]: pgmap v309: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:01.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[38106]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T05:27:01.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/279367800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi136-74781-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[29560]: pgmap v309: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[29560]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T05:27:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/279367800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi136-74781-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:01 smithi164 ceph-mon[33554]: pgmap v309: 268 pgs: 40 unknown, 228 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:01 smithi164 ceph-mon[33554]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T05:27:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/279367800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi136-74781-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:02.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:27:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:27:02.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/279367800' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi136-74781-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[38106]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/279367800' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi136-74781-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[29560]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:02.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:02.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:02.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/279367800' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi136-74781-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:02 smithi164 ceph-mon[33554]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T05:27:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: pgmap v312: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]': finished 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]: dispatch 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]: dispatch 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]': finished 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T05:27:03.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: pgmap v312: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]': finished 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]': finished 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: pgmap v312: 292 pgs: 32 creating+peering, 260 active+clean; 4.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-29"}]': finished 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]: dispatch 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1280345291' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]: dispatch 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]: dispatch 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-29", "mode": "writeback"}]': finished 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi136-74761-37"}]': finished 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:03.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:03.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:05.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:05 smithi164 ceph-mon[33554]: pgmap v315: 260 pgs: 260 active+clean; 4.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T05:27:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:05 smithi164 ceph-mon[33554]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T05:27:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[38106]: pgmap v315: 260 pgs: 260 active+clean; 4.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T05:27:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[38106]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T05:27:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[29560]: pgmap v315: 260 pgs: 260 active+clean; 4.6 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T05:27:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi136-74761-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[29560]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T05:27:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:06.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:06.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:06 smithi164 ceph-mon[33554]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T05:27:06.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:06.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:06.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:06.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:06 smithi136 ceph-mon[38106]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T05:27:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:06 smithi136 ceph-mon[29560]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T05:27:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:07.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:27:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[38106]: pgmap v318: 268 pgs: 8 creating+peering, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[38106]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[29560]: pgmap v318: 268 pgs: 8 creating+peering, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T05:27:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[29560]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T05:27:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:07.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:07 smithi164 ceph-mon[33554]: pgmap v318: 268 pgs: 8 creating+peering, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T05:27:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:07 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi136-74761-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:07 smithi164 ceph-mon[33554]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T05:27:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]: dispatch 2024-03-26T05:27:08.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:08 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]': finished 2024-03-26T05:27:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:08 smithi164 ceph-mon[33554]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T05:27:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:08 smithi164 ceph-mon[33554]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T05:27:09.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]': finished 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[29560]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[29560]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-29"}]': finished 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[38106]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:09.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:09.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:08 smithi136 ceph-mon[38106]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: pgmap v321: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:09.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:09.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:09.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:09.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: pgmap v321: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T05:27:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: pgmap v321: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/437304986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-46"}]': finished 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1829600809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:10.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:10.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:10.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:11.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: pgmap v324: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi136-74761-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:12.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: pgmap v324: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi136-74761-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: pgmap v324: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi136-74761-38"}]': finished 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi136-74781-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi136-74761-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:12.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:27:12.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi136-74761-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-mon[33554]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T05:27:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi136-74761-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:13.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi136-74761-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[38106]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi136-74761-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi136-74761-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:13.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[29560]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T05:27:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi136-74761-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi136-74781-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:13.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: pgmap v327: 260 pgs: 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]': finished 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]: dispatch 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]: dispatch 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi136-74761-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]': finished 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: pgmap v327: 260 pgs: 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:14.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]': finished 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]: dispatch 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]: dispatch 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi136-74761-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]': finished 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: pgmap v327: 260 pgs: 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-31"}]': finished 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]: dispatch 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]: dispatch 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi136-74761-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-31", "mode": "writeback"}]': finished 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:15.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:15 smithi164 ceph-mon[33554]: pgmap v330: 276 pgs: 16 unknown, 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:15.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:15 smithi164 ceph-mon[33554]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T05:27:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:15.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[38106]: pgmap v330: 276 pgs: 16 unknown, 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:15.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:15.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[38106]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[29560]: pgmap v330: 276 pgs: 16 unknown, 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[29560]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]: dispatch 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:16.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]': finished 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T05:27:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]': finished 2024-03-26T05:27:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T05:27:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-31"}]': finished 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3165087189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]: dispatch 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi136-74781-47"}]': finished 2024-03-26T05:27:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T05:27:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]: dispatch 2024-03-26T05:27:17.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:27:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:27:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: pgmap v333: 260 pgs: 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:27:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:17.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:18.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: pgmap v333: 260 pgs: 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: pgmap v333: 260 pgs: 260 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/848653616' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi136-74761-39"}]': finished 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi136-74781-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:18.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:18.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:18.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:18.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:19.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: pgmap v336: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T05:27:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:19 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: pgmap v336: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:19.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: pgmap v336: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi136-74761-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T05:27:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:19.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:19.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:19 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:20.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:20 smithi164 ceph-mon[33554]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T05:27:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:20.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:20.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[29560]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T05:27:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi136-74781-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[38106]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T05:27:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:21.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[29560]: pgmap v339: 268 pgs: 40 unknown, 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]': finished 2024-03-26T05:27:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[29560]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T05:27:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]: dispatch 2024-03-26T05:27:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]: dispatch 2024-03-26T05:27:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[38106]: pgmap v339: 268 pgs: 40 unknown, 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]': finished 2024-03-26T05:27:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[38106]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T05:27:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]: dispatch 2024-03-26T05:27:21.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]: dispatch 2024-03-26T05:27:21.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:21 smithi164 ceph-mon[33554]: pgmap v339: 268 pgs: 40 unknown, 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:21.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi136-74761-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:21.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-33"}]': finished 2024-03-26T05:27:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:21 smithi164 ceph-mon[33554]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T05:27:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]: dispatch 2024-03-26T05:27:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]: dispatch 2024-03-26T05:27:22.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]': finished 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]': finished 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-33", "mode": "writeback"}]': finished 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:22.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:22.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T05:27:22.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3374191117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]: dispatch 2024-03-26T05:27:22.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:22.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]: dispatch 2024-03-26T05:27:22.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: pgmap v342: 268 pgs: 8 creating+peering, 260 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]': finished 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: pgmap v342: 268 pgs: 8 creating+peering, 260 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]': finished 2024-03-26T05:27:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T05:27:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:23.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: pgmap v342: 268 pgs: 8 creating+peering, 260 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-33"}]': finished 2024-03-26T05:27:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi136-74781-48"}]': finished 2024-03-26T05:27:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/94832392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T05:27:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]: dispatch 2024-03-26T05:27:24.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi136-74761-40"}]': finished 2024-03-26T05:27:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi136-74781-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T05:27:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:25.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: pgmap v345: 260 pgs: 260 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T05:27:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: pgmap v345: 260 pgs: 260 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T05:27:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: pgmap v345: 260 pgs: 260 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi136-74761-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:26.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:26.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T05:27:26.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[38106]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T05:27:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:26 smithi136 ceph-mon[29560]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T05:27:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi136-74781-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:26 smithi164 ceph-mon[33554]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T05:27:27.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:27:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:27:28.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[38106]: pgmap v348: 260 pgs: 27 creating+peering, 5 unknown, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[38106]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T05:27:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[29560]: pgmap v348: 260 pgs: 27 creating+peering, 5 unknown, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[29560]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T05:27:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:27 smithi164 ceph-mon[33554]: pgmap v348: 260 pgs: 27 creating+peering, 5 unknown, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi136-74761-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:27 smithi164 ceph-mon[33554]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T05:27:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:29.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[38106]: pgmap v351: 276 pgs: 16 unknown, 32 creating+peering, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[38106]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]: dispatch 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]: dispatch 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[29560]: pgmap v351: 276 pgs: 16 unknown, 32 creating+peering, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[29560]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]: dispatch 2024-03-26T05:27:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]: dispatch 2024-03-26T05:27:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:29.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:29 smithi164 ceph-mon[33554]: pgmap v351: 276 pgs: 16 unknown, 32 creating+peering, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:27:29.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:29 smithi164 ceph-mon[33554]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T05:27:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]: dispatch 2024-03-26T05:27:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]: dispatch 2024-03-26T05:27:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]': finished 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:30.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]': finished 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:30.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:30.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:30.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-35", "mode": "writeback"}]': finished 2024-03-26T05:27:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T05:27:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1398998827' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]: dispatch 2024-03-26T05:27:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi136-74781-49"}]': finished 2024-03-26T05:27:31.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: pgmap v354: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi136-74781-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi136-74781-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T05:27:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi136-74781-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: pgmap v354: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi136-74781-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi136-74781-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T05:27:31.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi136-74781-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: pgmap v354: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/367020672' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]: dispatch 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi136-74781-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi136-74761-41"}]': finished 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi136-74781-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T05:27:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi136-74781-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:32.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:27:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:27:33.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi136-74761-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi136-74761-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi136-74761-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi136-74761-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:33.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi136-74761-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi136-74761-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:34.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: pgmap v357: 260 pgs: 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi136-74761-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi136-74781-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[38106]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: pgmap v357: 260 pgs: 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:27:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T05:27:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi136-74761-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi136-74781-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:33 smithi136 ceph-mon[29560]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T05:27:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: pgmap v357: 260 pgs: 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi136-74761-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi136-74781-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:33 smithi164 ceph-mon[33554]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T05:27:35.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[38106]: pgmap v360: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:27:35.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi136-74761-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:35.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[38106]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T05:27:35.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:35.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:35.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[29560]: pgmap v360: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:27:35.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi136-74761-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:35.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[29560]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T05:27:35.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:35.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:35.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:35 smithi164 ceph-mon[33554]: pgmap v360: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:27:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi136-74761-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:35 smithi164 ceph-mon[33554]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T05:27:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:36.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[38106]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T05:27:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:36.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:36.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:36.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[29560]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T05:27:36.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:36.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:36.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:36 smithi164 ceph-mon[33554]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T05:27:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]: dispatch 2024-03-26T05:27:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:37.495 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:27:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:27:37.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[38106]: pgmap v363: 268 pgs: 4 creating+activating, 1 unknown, 263 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:27:37.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:37.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[38106]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[29560]: pgmap v363: 268 pgs: 4 creating+activating, 1 unknown, 263 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[29560]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T05:27:37.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:37.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:37.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:37 smithi164 ceph-mon[33554]: pgmap v363: 268 pgs: 4 creating+activating, 1 unknown, 263 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:27:37.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:37 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:37.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-35"}]': finished 2024-03-26T05:27:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:37 smithi164 ceph-mon[33554]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T05:27:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]: dispatch 2024-03-26T05:27:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6977 ms) 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7015 ms) 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6995 ms) 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7013 ms) 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7025 ms) 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-26T05:27:38.525 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7030 ms) 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7018 ms) 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7017 ms) 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6922 ms) 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7010 ms) 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6968 ms) 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7024 ms) 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-26T05:27:38.526 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6991 ms) 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7021 ms) 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7016 ms) 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7035 ms) 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112077 ms total) 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [----------] Global test environment tear-down 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (206770 ms total) 2024-03-26T05:27:38.527 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-26T05:27:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:38.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[38106]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T05:27:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:38.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[29560]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T05:27:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:38.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:38 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:38.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1180010146' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi136-74781-50"}]': finished 2024-03-26T05:27:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:38 smithi164 ceph-mon[33554]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T05:27:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]: dispatch 2024-03-26T05:27:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:38 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:39.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[38106]: pgmap v366: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[38106]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3127634745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[29560]: pgmap v366: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[29560]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3127634745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:39 smithi164 ceph-mon[33554]: pgmap v366: 228 pgs: 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:27:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3194918257' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi136-74761-42"}]': finished 2024-03-26T05:27:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:39 smithi164 ceph-mon[33554]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T05:27:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3127634745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:40.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3127634745' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:40.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:40.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[38106]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T05:27:40.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:40.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:40.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3127634745' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:40.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:40.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[29560]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T05:27:40.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:40.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:40.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3127634745' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi136-74781-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:40.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:40 smithi164 ceph-mon[33554]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T05:27:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:41.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: pgmap v369: 292 pgs: 64 unknown, 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: pgmap v369: 292 pgs: 64 unknown, 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:41.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: pgmap v369: 292 pgs: 64 unknown, 228 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:42.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:27:43.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]': finished 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]: dispatch 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]: dispatch 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]': finished 2024-03-26T05:27:43.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]': finished 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]: dispatch 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]: dispatch 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:43.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]': finished 2024-03-26T05:27:43.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-37"}]': finished 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]: dispatch 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]: dispatch 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-37", "mode": "writeback"}]': finished 2024-03-26T05:27:44.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: pgmap v372: 260 pgs: 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: pgmap v372: 260 pgs: 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: pgmap v372: 260 pgs: 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi136-74781-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]: dispatch 2024-03-26T05:27:45.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:44 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:45.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]': finished 2024-03-26T05:27:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:44 smithi136 ceph-mon[38106]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T05:27:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:44 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]': finished 2024-03-26T05:27:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:44 smithi136 ceph-mon[29560]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T05:27:45.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:44 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-37"}]': finished 2024-03-26T05:27:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:44 smithi164 ceph-mon[33554]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T05:27:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:45 smithi136 ceph-mon[38106]: pgmap v375: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:45 smithi136 ceph-mon[38106]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T05:27:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:45 smithi136 ceph-mon[29560]: pgmap v375: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:45 smithi136 ceph-mon[29560]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T05:27:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:46.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:45 smithi164 ceph-mon[33554]: pgmap v375: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:46.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:45 smithi164 ceph-mon[33554]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T05:27:46.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:27:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:27:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[29560]: pgmap v378: 228 pgs: 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:27:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[29560]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T05:27:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[38106]: pgmap v378: 228 pgs: 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:27:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[38106]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T05:27:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:47.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:47 smithi164 ceph-mon[33554]: pgmap v378: 228 pgs: 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:27:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:47 smithi164 ceph-mon[33554]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T05:27:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/4037030729' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]: dispatch 2024-03-26T05:27:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:48.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi136-74781-52"}]': finished 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T05:27:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:48.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:48.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:48.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:49.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:49 smithi164 ceph-mon[33554]: pgmap v381: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:27:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]': finished 2024-03-26T05:27:50.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:49 smithi136 ceph-mon[29560]: pgmap v381: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:27:50.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]': finished 2024-03-26T05:27:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:49 smithi136 ceph-mon[38106]: pgmap v381: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:27:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-39"}]': finished 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]: dispatch 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]: dispatch 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]': finished 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T05:27:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:50.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:50.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:27:50.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:27:50.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:50 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]: dispatch 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]: dispatch 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]': finished 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]: dispatch 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]: dispatch 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi136-74781-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-39", "mode": "writeback"}]': finished 2024-03-26T05:27:51.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T05:27:51.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:51.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:27:51.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:27:51.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:27:51.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:50 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:27:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:51 smithi164 ceph-mon[33554]: pgmap v384: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:51 smithi164 ceph-mon[33554]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T05:27:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[29560]: pgmap v384: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:52.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[29560]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T05:27:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[38106]: pgmap v384: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:27:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[38106]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T05:27:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]: dispatch 2024-03-26T05:27:52.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:27:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:27:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:52 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]': finished 2024-03-26T05:27:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:52 smithi164 ceph-mon[33554]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T05:27:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:53.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]': finished 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[38106]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-39"}]': finished 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[29560]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:53.464 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: seed 75071 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (280 ms) 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (15723 ms) 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (9 ms) 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (16012 ms total) 2024-03-26T05:27:53.465 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (8656 ms) 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8665 ms) 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9334 ms) 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: my_snaps [3] 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: my_snaps [4,3] 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: promoting some heads 2024-03-26T05:27:53.466 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: done waiting 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (45988 ms) 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10003 ms) 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8022 ms) 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8015 ms) 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8039 ms) 2024-03-26T05:27:53.467 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10272 ms) 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8682 ms) 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10021 ms) 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (11983 ms) 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7995 ms) 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8024 ms) 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-26T05:27:53.468 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13037 ms) 2024-03-26T05:27:53.469 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-26T05:27:53.469 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7945 ms) 2024-03-26T05:27:53.469 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-26T05:27:53.469 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8014 ms) 2024-03-26T05:27:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[38106]: pgmap v387: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[38106]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[29560]: pgmap v387: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[29560]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T05:27:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:54.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:53 smithi164 ceph-mon[33554]: pgmap v387: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 120 op/s 2024-03-26T05:27:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:27:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:53 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:53 smithi164 ceph-mon[33554]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T05:27:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1828934356' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]: dispatch 2024-03-26T05:27:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[38106]: pgmap v390: 228 pgs: 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T05:27:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[38106]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T05:27:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi136-74781-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:55.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[29560]: pgmap v390: 228 pgs: 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T05:27:55.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:55.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[29560]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T05:27:55.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:55.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:55.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:55.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:55.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi136-74781-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:55.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:55 smithi164 ceph-mon[33554]: pgmap v390: 228 pgs: 228 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T05:27:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi136-74781-53"}]': finished 2024-03-26T05:27:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:55 smithi164 ceph-mon[33554]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T05:27:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:27:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi136-74781-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:27:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi136-74781-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:56 smithi164 ceph-mon[33554]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T05:27:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi136-74781-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:57.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi136-74781-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[29560]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi136-74781-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi136-74781-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[38106]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T05:27:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi136-74781-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:27:57.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:57.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:27:57.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:27:57.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:27:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: pgmap v393: 260 pgs: 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi136-74781-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T05:27:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:27:58.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: pgmap v393: 260 pgs: 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi136-74781-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: pgmap v393: 260 pgs: 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:27:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:27:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T05:27:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:27:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:27:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi136-74781-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:27:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:27:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:27:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T05:27:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:27:58.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:58 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:27:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:27:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:58 smithi164 ceph-mon[33554]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T05:27:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:27:59.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[38106]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[29560]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T05:27:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:27:59.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:59 smithi164 ceph-mon[33554]: pgmap v396: 268 pgs: 8 unknown, 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:27:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:27:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:59 smithi164 ceph-mon[33554]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T05:27:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:27:59 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:28:00.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:59 smithi136 ceph-mon[38106]: pgmap v396: 268 pgs: 8 unknown, 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:28:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:59 smithi136 ceph-mon[38106]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T05:28:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:27:59 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:28:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:59 smithi136 ceph-mon[29560]: pgmap v396: 268 pgs: 8 unknown, 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:28:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:59 smithi136 ceph-mon[29560]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T05:28:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:27:59 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[29560]: pgmap v399: 260 pgs: 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[29560]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]: dispatch 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]: dispatch 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[38106]: pgmap v399: 260 pgs: 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:28:01.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[38106]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T05:28:01.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:28:01.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:01.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:01.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]: dispatch 2024-03-26T05:28:01.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]: dispatch 2024-03-26T05:28:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:01 smithi164 ceph-mon[33554]: pgmap v399: 260 pgs: 5 creating+activating, 255 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:28:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:01 smithi164 ceph-mon[33554]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T05:28:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]: dispatch 2024-03-26T05:28:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]: dispatch 2024-03-26T05:28:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]: dispatch 2024-03-26T05:28:02.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:28:02.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]': finished 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]': finished 2024-03-26T05:28:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T05:28:02.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:02.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2209114030' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi136-74781-54"}]': finished 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-41"}]': finished 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:03.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[38106]: pgmap v402: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:03.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[38106]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[29560]: pgmap v402: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[29560]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T05:28:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:03.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:03 smithi164 ceph-mon[33554]: pgmap v402: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi136-74781-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:03 smithi164 ceph-mon[33554]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T05:28:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:04 smithi136 ceph-mon[29560]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T05:28:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:04.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:04 smithi136 ceph-mon[38106]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T05:28:04.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:04.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:04 smithi164 ceph-mon[33554]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T05:28:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:05.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[38106]: pgmap v405: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[38106]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T05:28:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi136-75071-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T05:28:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[29560]: pgmap v405: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[29560]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T05:28:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi136-75071-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T05:28:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:05.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:05 smithi164 ceph-mon[33554]: pgmap v405: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:05 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi136-74781-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:05 smithi164 ceph-mon[33554]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T05:28:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi136-75071-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T05:28:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:06.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:06 smithi136 ceph-mon[38106]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T05:28:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T05:28:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T05:28:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:06 smithi136 ceph-mon[29560]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T05:28:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T05:28:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T05:28:06.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:06.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:06 smithi164 ceph-mon[33554]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T05:28:06.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T05:28:06.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T05:28:07.527 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:28:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:28:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[29560]: pgmap v408: 268 pgs: 3 unknown, 265 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T05:28:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[29560]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T05:28:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[38106]: pgmap v408: 268 pgs: 3 unknown, 265 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T05:28:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[38106]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T05:28:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:07.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:07 smithi164 ceph-mon[33554]: pgmap v408: 268 pgs: 3 unknown, 265 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T05:28:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:07 smithi164 ceph-mon[33554]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T05:28:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: ok, hit_set contains 260:602f83fe:::foo:head 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9040 ms) 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg_num = 32 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 0 ls 1711430889,0 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 1 ls 1711430889,0 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 2 ls 1711430889,0 2024-03-26T05:28:08.758 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 3 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 4 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 5 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 6 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 7 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 8 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 9 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 10 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 11 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 12 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 13 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 14 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 15 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 16 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 17 ls 1711430889,0 2024-03-26T05:28:08.759 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 18 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 19 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 20 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 21 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 22 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 23 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 24 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 25 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 26 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 27 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 28 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 29 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 30 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg 31 ls 1711430889,0 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: pg_num = 32 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:6cac518f:::0:head 2024-03-26T05:28:08.760 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:02547ec2:::1:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f905c69b:::2:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:cfc208b3:::3:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d83876eb:::4:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b29083e3:::5:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c4fdafeb:::6:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:5c6b0b28:::7:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:bd63b0f1:::8:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:e960b815:::9:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:52ea6a34:::10:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:89d3ae78:::11:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:de5d7c5f:::12:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:566253c9:::13:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:62a1935d:::14:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:863748b0:::15:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:3958e169:::16:head 2024-03-26T05:28:08.761 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:4d4dabf9:::17:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:8391935d:::18:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:28883081:::19:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:69259c59:::20:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:4bdb80b7:::21:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a11c5d71:::22:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:271af37b:::23:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:95b121be:::24:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:58d1031b:::25:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:0a050783:::26:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c709704c:::27:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:cbe56eaf:::28:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:86b4b162:::29:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:70d89383:::30:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:dd450c7c:::31:head 2024-03-26T05:28:08.762 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:6d5729b1:::32:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c388f3fb:::33:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:56cfea31:::34:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:9dbc1bf7:::35:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:40b74ccd:::36:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:4d5aaf42:::37:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:920f362c:::38:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:6cc53222:::39:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:9cad833f:::40:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:1ea84d41:::41:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c4480ef6:::42:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a694361e:::43:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d1bd33e9:::44:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:ddc2cd5d:::45:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:2b782207:::46:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:7b187fca:::47:head 2024-03-26T05:28:08.763 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:90ecdf6f:::48:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a5ed95fe:::49:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:ea0eaa55:::50:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f33ef17b:::51:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a0d1b2f6:::52:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:60c5229e:::53:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:edcbc575:::54:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:102cf253:::55:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:efb7fb0b:::56:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:50d0a326:::57:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d4dc5daf:::58:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:3a130462:::59:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:ec87ed71:::60:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d5bc9454:::61:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:3ddfe313:::62:head 2024-03-26T05:28:08.764 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:7c2816b9:::63:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:47e00e4d:::64:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c6410c18:::65:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b48ed237:::66:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:cd63ad31:::67:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b179e92b:::68:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:0d9f741a:::69:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:6d3352ae:::70:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c6d5c19e:::71:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:bc4729c3:::72:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:77e930b9:::73:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:0abeecfd:::74:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b7c37e15:::75:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b6378398:::76:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:02bd68de:::77:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:cc795d2d:::78:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:630d4fea:::79:head 2024-03-26T05:28:08.765 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:e0d29ef5:::80:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:fd6f13d2:::81:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:606461d5:::82:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:eadbdc43:::83:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:8761d0bb:::84:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:9ef0186f:::85:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:e0d41294:::86:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:961de695:::87:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:1423148f:::88:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:633a8fa2:::89:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a8653809:::90:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:3dac8b33:::91:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:35aad435:::92:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f6dcc343:::93:head 2024-03-26T05:28:08.766 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:dbbdad87:::94:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:1cb48ce0:::95:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:03cd461c:::96:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:17a4ea99:::97:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:9993c9a7:::98:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:6394211c:::99:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:94c7ae57:::100:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:6fdee5bb:::101:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:9a477fd1:::102:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:eb850916:::103:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:affc56b9:::104:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b42dc814:::105:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f319f8f0:::106:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:9a40b9de:::107:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:8b524f28:::108:head 2024-03-26T05:28:08.767 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:e3de589f:::109:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:90f90a5b:::110:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a7b4f1d7:::111:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:af51766e:::112:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b6f90bd1:::113:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:e0261208:::114:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c9569ef7:::115:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:61bebe50:::116:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:fe93412b:::117:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d3d38bee:::118:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:3100ba0c:::119:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d0560ada:::120:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f0ea8b35:::121:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:766f231a:::122:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a07a2582:::123:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:bd7c6b3a:::124:head 2024-03-26T05:28:08.768 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:fb2ddaff:::125:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:4408e1fe:::126:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:ee1df7a7:::127:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c3002909:::128:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:4f48ffa9:::129:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:edf38733:::130:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c08425c0:::131:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:5f902d98:::132:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:41ea2c93:::133:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:813cee13:::134:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:0131818d:::135:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:26ba5a85:::136:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:381b8a5a:::137:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:28797e47:::138:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:bfca7f22:::139:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:36807075:::140:head 2024-03-26T05:28:08.769 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:80b03975:::141:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:5c15709b:::142:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f39ea15e:::143:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:ea992956:::144:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:48887b1c:::145:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:9f24a9dd:::146:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:987f100b:::147:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d2dd3581:::148:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:7fed1808:::149:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c80b70e9:::150:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:85ed90f9:::151:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:36428b24:::152:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d044c34a:::153:head 2024-03-26T05:28:08.770 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:7c18bf58:::154:head 2024-03-26T05:28:08.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d1c21232:::155:head 2024-03-26T05:28:08.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a7a3c575:::156:head 2024-03-26T05:28:08.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:87da0633:::157:head 2024-03-26T05:28:08.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d5ac3822:::158:head 2024-03-26T05:28:08.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:3f20522d:::159:head 2024-03-26T05:28:08.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:6ca26563:::160:head 2024-03-26T05:28:08.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:532ce135:::161:head 2024-03-26T05:28:08.771 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:c78863e6:::162:head 2024-03-26T05:28:08.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[38106]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[29560]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T05:28:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T05:28:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T05:28:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T05:28:08.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:28:08.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:08.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:08 smithi164 ceph-mon[33554]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T05:28:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T05:28:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2642222612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T05:28:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]: dispatch 2024-03-26T05:28:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T05:28:09.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: pgmap v411: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi136-75071-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]: dispatch 2024-03-26T05:28:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]: dispatch 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: pgmap v411: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi136-75071-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T05:28:09.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:09.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:09.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]: dispatch 2024-03-26T05:28:09.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]: dispatch 2024-03-26T05:28:09.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: pgmap v411: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:09.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi136-74781-55"}]': finished 2024-03-26T05:28:09.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T05:28:09.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:09.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:09.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:09.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi136-75071-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T05:28:09.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:09.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:09.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]: dispatch 2024-03-26T05:28:09.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]: dispatch 2024-03-26T05:28:10.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]': finished 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[29560]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]': finished 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[38106]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:10.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi136-74781-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-43"}]': finished 2024-03-26T05:28:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:10 smithi164 ceph-mon[33554]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T05:28:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:10.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:11 smithi136 ceph-mon[38106]: pgmap v414: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:11 smithi136 ceph-mon[38106]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T05:28:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:11 smithi136 ceph-mon[29560]: pgmap v414: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:11 smithi136 ceph-mon[29560]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T05:28:11.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:11 smithi164 ceph-mon[33554]: pgmap v414: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:11 smithi164 ceph-mon[33554]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T05:28:12.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:28:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[29560]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[38106]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:12.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:12.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi136-74781-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:12 smithi164 ceph-mon[33554]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T05:28:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:13 smithi164 ceph-mon[33554]: pgmap v417: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:13 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:13 smithi164 ceph-mon[33554]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T05:28:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[38106]: pgmap v417: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[38106]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T05:28:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[29560]: pgmap v417: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[29560]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T05:28:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:14.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:28:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:28:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2482003541' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:28:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]: dispatch 2024-03-26T05:28:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:15 smithi164 ceph-mon[33554]: pgmap v420: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:28:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:15 smithi164 ceph-mon[33554]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T05:28:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:16.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[29560]: pgmap v420: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[29560]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[38106]: pgmap v420: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi136-74781-56"}]': finished 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[38106]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T05:28:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:28:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:28:16.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:16.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi136-74781-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:17.577 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:28:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:28:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:28:18.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:17 smithi136 ceph-mon[38106]: pgmap v423: 260 pgs: 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:28:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:17 smithi136 ceph-mon[29560]: pgmap v423: 260 pgs: 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:28:18.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:17 smithi164 ceph-mon[33554]: pgmap v423: 260 pgs: 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:28:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:18 smithi164 ceph-mon[33554]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T05:28:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:18 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:18 smithi164 ceph-mon[33554]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T05:28:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:18 smithi136 ceph-mon[38106]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T05:28:19.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:18 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:18 smithi136 ceph-mon[38106]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T05:28:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:18 smithi136 ceph-mon[29560]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T05:28:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:18 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi136-74781-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:18 smithi136 ceph-mon[29560]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T05:28:20.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:19 smithi164 ceph-mon[33554]: pgmap v426: 260 pgs: 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:19 smithi164 ceph-mon[33554]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T05:28:20.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:19 smithi136 ceph-mon[38106]: pgmap v426: 260 pgs: 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:20.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:19 smithi136 ceph-mon[38106]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T05:28:20.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:19 smithi136 ceph-mon[29560]: pgmap v426: 260 pgs: 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:20.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:19 smithi136 ceph-mon[29560]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T05:28:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:21 smithi164 ceph-mon[33554]: pgmap v429: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:21 smithi164 ceph-mon[33554]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T05:28:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:21 smithi136 ceph-mon[38106]: pgmap v429: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:21 smithi136 ceph-mon[38106]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T05:28:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:21 smithi136 ceph-mon[29560]: pgmap v429: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:21 smithi136 ceph-mon[29560]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T05:28:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:22.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:28:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:28:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:22 smithi164 ceph-mon[33554]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T05:28:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:23.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:23.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:22 smithi136 ceph-mon[38106]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T05:28:23.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:23.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:22 smithi136 ceph-mon[29560]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T05:28:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1495104609' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]: dispatch 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: pgmap v432: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:23.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: pgmap v432: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: pgmap v432: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi136-74781-57"}]': finished 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:25.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:24 smithi164 ceph-mon[33554]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T05:28:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:25.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:24 smithi136 ceph-mon[38106]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T05:28:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi136-74781-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:24 smithi136 ceph-mon[29560]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T05:28:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:26.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:25 smithi164 ceph-mon[33554]: pgmap v435: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:28:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:25 smithi164 ceph-mon[33554]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T05:28:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:25 smithi164 ceph-mon[33554]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T05:28:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:25 smithi136 ceph-mon[38106]: pgmap v435: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:28:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:25 smithi136 ceph-mon[38106]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T05:28:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:25 smithi136 ceph-mon[38106]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T05:28:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:25 smithi136 ceph-mon[29560]: pgmap v435: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:28:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:25 smithi136 ceph-mon[29560]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T05:28:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi136-74781-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:25 smithi136 ceph-mon[29560]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T05:28:27.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:28:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:28:27.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:27 smithi164 ceph-mon[33554]: pgmap v438: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:28:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:27 smithi164 ceph-mon[33554]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T05:28:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:27 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-mon[38106]: pgmap v438: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:28:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-mon[38106]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T05:28:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-mon[29560]: pgmap v438: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:28:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-mon[29560]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T05:28:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:28 smithi164 ceph-mon[33554]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T05:28:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:29.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:28 smithi136 ceph-mon[38106]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T05:28:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:28 smithi136 ceph-mon[29560]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T05:28:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:29.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:29 smithi164 ceph-mon[33554]: pgmap v441: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:28:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:29 smithi164 ceph-mon[33554]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T05:28:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:30.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[38106]: pgmap v441: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[38106]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[29560]: pgmap v441: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[29560]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2950126745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]: dispatch 2024-03-26T05:28:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T05:28:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:30.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:30.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:30.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]: dispatch 2024-03-26T05:28:30.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]: dispatch 2024-03-26T05:28:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T05:28:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:31.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:31.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:31.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]: dispatch 2024-03-26T05:28:31.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]: dispatch 2024-03-26T05:28:31.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi136-74781-58"}]': finished 2024-03-26T05:28:31.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T05:28:31.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:31.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:31.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:31.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:31.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:31.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]: dispatch 2024-03-26T05:28:31.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]: dispatch 2024-03-26T05:28:32.182 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:31 smithi164 ceph-mon[33554]: pgmap v444: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:32.182 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:32.183 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]': finished 2024-03-26T05:28:32.183 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:31 smithi164 ceph-mon[33554]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T05:28:32.183 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi136-74781-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[38106]: pgmap v444: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]': finished 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[38106]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi136-74781-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[29560]: pgmap v444: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-45"}]': finished 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[29560]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T05:28:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi136-74781-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:32.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:28:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:28:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:32 smithi164 ceph-mon[33554]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T05:28:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:32 smithi164 ceph-mon[33554]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:32 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi136-74781-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:32 smithi164 ceph-mon[33554]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T05:28:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:33.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[38106]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T05:28:33.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[38106]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi136-74781-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[38106]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[29560]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[29560]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi136-74781-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[29560]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T05:28:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:33.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:33 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:33 smithi164 ceph-mon[33554]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T05:28:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[38106]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T05:28:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[29560]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T05:28:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:35.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:35 smithi164 ceph-mon[33554]: pgmap v450: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:35 smithi164 ceph-mon[33554]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T05:28:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:36.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[38106]: pgmap v450: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[38106]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[29560]: pgmap v450: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[29560]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T05:28:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:36.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:36.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:36.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]': finished 2024-03-26T05:28:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:36 smithi164 ceph-mon[33554]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T05:28:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]: dispatch 2024-03-26T05:28:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]: dispatch 2024-03-26T05:28:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:36 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:28:36.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]': finished 2024-03-26T05:28:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[38106]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T05:28:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]: dispatch 2024-03-26T05:28:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]: dispatch 2024-03-26T05:28:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:28:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:37.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:37.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-47"}]': finished 2024-03-26T05:28:37.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[29560]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T05:28:37.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]: dispatch 2024-03-26T05:28:37.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]: dispatch 2024-03-26T05:28:37.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]: dispatch 2024-03-26T05:28:37.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:28:37.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2462284786' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi136-74781-59"}]': finished 2024-03-26T05:28:37.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:28:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: pgmap v453: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]': finished 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3681926740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: pgmap v453: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]': finished 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3681926740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: pgmap v453: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-47", "mode": "writeback"}]': finished 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:28:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3681926740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:39.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:7462ddf6:::1.RoundTripAppendPP (3008 ms) 2024-03-26T05:28:39.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-26T05:28:39.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3239 ms) 2024-03-26T05:28:39.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-26T05:28:39.673 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2771 ms) 2024-03-26T05:28:39.673 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-26T05:28:39.673 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3026 ms) 2024-03-26T05:28:39.673 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-26T05:28:39.673 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: setting pool EIO 2024-03-26T05:28:39.673 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: max_success 99, min_failed 100 2024-03-26T05:28:39.673 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4006 ms) 2024-03-26T05:28:39.673 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3011 ms) 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (137746 ms total) 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3004 ms) 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9028 ms) 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-26T05:28:39.674 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14941 ms) 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3005 ms) 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29978 ms total) 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3030 ms) 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3030 ms total) 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13938 ms) 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7025 ms) 2024-03-26T05:28:39.675 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7014 ms) 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7266 ms) 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2782 ms) 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6921 ms) 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7017 ms) 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7014 ms) 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7046 ms) 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7049 ms) 2024-03-26T05:28:39.676 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-26T05:28:39.677 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7025 ms) 2024-03-26T05:28:39.677 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-26T05:28:39.677 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7034 ms) 2024-03-26T05:28:39.677 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-26T05:28:39.677 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7027 ms) 2024-03-26T05:28:39.677 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-03-26T05:28:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:39 smithi164 ceph-mon[33554]: pgmap v456: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:28:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:28:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3681926740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:39 smithi164 ceph-mon[33554]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T05:28:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:39 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:40.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[38106]: pgmap v456: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3681926740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[38106]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[29560]: pgmap v456: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3681926740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi136-74781-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[29560]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T05:28:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:40.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:28:40.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:39 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi136-74781-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi136-74781-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi136-74781-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:41.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:28:42.191 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:41 smithi164 ceph-mon[33554]: pgmap v459: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:42.191 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi136-74781-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:42.191 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:28:42.191 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:41 smithi164 ceph-mon[33554]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T05:28:42.191 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi136-74781-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:42.191 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:28:42.191 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:28:42.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[38106]: pgmap v459: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:42.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi136-74781-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:42.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[38106]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi136-74781-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[29560]: pgmap v459: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi136-74781-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[29560]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi136-74781-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:28:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:28:42.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: pgmap v462: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi136-74781-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:43.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: pgmap v462: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi136-74781-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:28:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: pgmap v462: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi136-74781-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:44.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:28:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:44 smithi164 ceph-mon[33554]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T05:28:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:28:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:44 smithi136 ceph-mon[38106]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T05:28:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:28:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:44 smithi136 ceph-mon[29560]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T05:28:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:45.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: pgmap v465: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]': finished 2024-03-26T05:28:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T05:28:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:46.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: pgmap v465: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]': finished 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: pgmap v465: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]': finished 2024-03-26T05:28:46.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T05:28:46.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:46.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:46.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:28:46.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:46.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-47"}]: dispatch 2024-03-26T05:28:46.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:46.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:46 smithi164 ceph-mon[33554]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T05:28:46.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:47.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:47.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:46 smithi136 ceph-mon[38106]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T05:28:47.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:47.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:47.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:46 smithi136 ceph-mon[29560]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T05:28:47.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]: dispatch 2024-03-26T05:28:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:28:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:28:47.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T05:28:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi136-74781-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T05:28:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi136-74781-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/77859021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi136-74781-61"}]': finished 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T05:28:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi136-74781-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:49.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:49.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:49.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:49.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:28:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: pgmap v471: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi136-74781-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]': finished 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]: dispatch 2024-03-26T05:28:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]: dispatch 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: pgmap v471: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi136-74781-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]': finished 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]: dispatch 2024-03-26T05:28:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]: dispatch 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: pgmap v471: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi136-74781-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-49"}]': finished 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]: dispatch 2024-03-26T05:28:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]: dispatch 2024-03-26T05:28:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:50 smithi164 ceph-mon[33554]: pgmap v474: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:51.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:50 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:51.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:50 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:28:51.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]': finished 2024-03-26T05:28:51.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:50 smithi164 ceph-mon[33554]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T05:28:51.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:50 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[38106]: pgmap v474: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]': finished 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[38106]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[29560]: pgmap v474: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:28:51.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-49", "mode": "readproxy"}]': finished 2024-03-26T05:28:51.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[29560]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T05:28:51.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:50 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:28:52.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:51 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:28:52.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:51 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:28:52.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:51 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:28:52.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:51 smithi164 ceph-mon[33554]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T05:28:52.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:28:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:28:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:28:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[38106]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T05:28:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:28:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:28:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:28:52.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[29560]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T05:28:52.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:52.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:28:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:28:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:53 smithi164 ceph-mon[33554]: pgmap v477: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:53 smithi164 ceph-mon[33554]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T05:28:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:54.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[38106]: pgmap v477: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[38106]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[29560]: pgmap v477: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[29560]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T05:28:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]: dispatch 2024-03-26T05:28:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[38106]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T05:28:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[29560]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T05:28:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:55.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3580934388' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi136-74781-62"}]': finished 2024-03-26T05:28:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:54 smithi164 ceph-mon[33554]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T05:28:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:28:56.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[38106]: pgmap v480: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[38106]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[29560]: pgmap v480: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[29560]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:55 smithi164 ceph-mon[33554]: pgmap v480: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:28:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi136-74781-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:28:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:55 smithi164 ceph-mon[33554]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T05:28:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:57.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:56 smithi136 ceph-mon[38106]: pgmap v483: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:57.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:56 smithi136 ceph-mon[38106]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T05:28:57.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:28:57.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:56 smithi136 ceph-mon[38106]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T05:28:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:56 smithi136 ceph-mon[29560]: pgmap v483: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:56 smithi136 ceph-mon[29560]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T05:28:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:28:57.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:56 smithi136 ceph-mon[29560]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T05:28:57.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:56 smithi164 ceph-mon[33554]: pgmap v483: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:56 smithi164 ceph-mon[33554]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T05:28:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi136-74781-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:28:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:56 smithi164 ceph-mon[33554]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T05:28:57.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:28:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:28:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:28:58.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:58 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:28:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:58 smithi164 ceph-mon[33554]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T05:28:58.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:58 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:28:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:58 smithi136 ceph-mon[38106]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T05:28:58.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:58 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:28:58.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:28:58.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:58 smithi136 ceph-mon[29560]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T05:28:59.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:59 smithi164 ceph-mon[33554]: pgmap v486: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:59 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:59 smithi164 ceph-mon[33554]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T05:28:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:59 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:28:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:59.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[38106]: pgmap v486: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[38106]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[29560]: pgmap v486: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[29560]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:28:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:28:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:29:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:29:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:00 smithi136 ceph-mon[38106]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T05:29:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:29:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:29:01.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:29:01.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:00 smithi136 ceph-mon[29560]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T05:29:01.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:29:01.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:29:01.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:29:01.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:00 smithi164 ceph-mon[33554]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T05:29:01.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3653226856' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:29:01.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]: dispatch 2024-03-26T05:29:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: pgmap v489: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:29:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T05:29:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: pgmap v489: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:02.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:02.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:02.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: pgmap v489: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi136-74781-63"}]': finished 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:02.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:02.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:02.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:29:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:29:03.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[38106]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[29560]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:03.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:03.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:03.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi136-74781-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:03.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:29:03.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:03.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:02 smithi164 ceph-mon[33554]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T05:29:03.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:03.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:03.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:03.779 INFO:tasks.workunit.client.0.smithi136.stdout:63:head 2024-03-26T05:29:03.779 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:5d165639:::164:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f43765fc:::165:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b4c720e9:::166:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:e694b040:::167:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:afa38db2:::168:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:77ba9f53:::169:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:87495034:::170:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:7c96bf0e:::171:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:dbe346cc:::172:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:e943ec24:::173:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f97a9c0c:::174:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:6f26e74d:::175:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:4f95e106:::176:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:0e6f2f8f:::177:head 2024-03-26T05:29:03.780 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:05db05f1:::178:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:38a78d66:::179:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:d095610b:::180:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:a1a9d709:::181:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:1e5d39db:::182:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:f7df4fb9:::183:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:03a7f161:::184:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:ba70721e:::185:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:28e5662d:::186:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:973d52de:::187:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:4303eb1c:::188:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:b990b48e:::189:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:29b8165b:::190:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:3547f197:::191:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:7e260936:::192:head 2024-03-26T05:29:03.781 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:1abec7b1:::193:head 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:10fdda93:::194:head 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:15817eea:::195:head 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:770bab57:::196:head 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:ed9e13e7:::197:head 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:71471a8f:::198:head 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: checking for 262:10fb1d02:::199:head 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8299 ms) 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,0 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: first is 1711430898 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,0 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,0 2024-03-26T05:29:03.782 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,1711430903,1711430904,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,1711430903,1711430904,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,1711430903,1711430904,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,1711430903,1711430904,1711430906,1711430907,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,1711430903,1711430904,1711430906,1711430907,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430898,1711430900,1711430901,1711430903,1711430904,1711430906,1711430907,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711430901,1711430903,1711430904,1711430906,1711430907,1711430909,1711430910,0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: first now 1711430901, trimmed 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21087 ms) 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: foo0 2024-03-26T05:29:03.783 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T05:29:03.784 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13793 ms) 2024-03-26T05:29:03.784 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: pgmap v492: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]': finished 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: pgmap v492: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:29:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]': finished 2024-03-26T05:29:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T05:29:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: pgmap v492: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]': finished 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-49"}]: dispatch 2024-03-26T05:29:05.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:05.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:04 smithi136 ceph-mon[38106]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T05:29:05.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:05.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:04 smithi136 ceph-mon[29560]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T05:29:05.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi136-74781-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:05.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:04 smithi164 ceph-mon[33554]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[38106]: pgmap v495: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[38106]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[29560]: pgmap v495: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[29560]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:05 smithi164 ceph-mon[33554]: pgmap v495: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:05 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:05 smithi164 ceph-mon[33554]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T05:29:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:29:07.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:07.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T05:29:07.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:07.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T05:29:07.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:07.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/231503904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]: dispatch 2024-03-26T05:29:07.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:29:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:29:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[38106]: pgmap v498: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:29:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]': finished 2024-03-26T05:29:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]: dispatch 2024-03-26T05:29:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[38106]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T05:29:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]: dispatch 2024-03-26T05:29:08.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[29560]: pgmap v498: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:29:08.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]': finished 2024-03-26T05:29:08.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:08.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]: dispatch 2024-03-26T05:29:08.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[29560]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T05:29:08.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]: dispatch 2024-03-26T05:29:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:07 smithi164 ceph-mon[33554]: pgmap v498: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:29:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-51"}]': finished 2024-03-26T05:29:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi136-74781-64"}]': finished 2024-03-26T05:29:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]: dispatch 2024-03-26T05:29:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:07 smithi164 ceph-mon[33554]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T05:29:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]: dispatch 2024-03-26T05:29:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi136-74781-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:29:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]': finished 2024-03-26T05:29:09.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi136-74781-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[38106]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi136-74781-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi136-74781-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]': finished 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi136-74781-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[29560]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T05:29:09.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi136-74781-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:09.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:09.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:09.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi136-74781-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:09.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:08 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:29:09.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-51", "mode": "writeback"}]': finished 2024-03-26T05:29:09.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi136-74781-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:09.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:08 smithi164 ceph-mon[33554]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T05:29:09.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi136-74781-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:10 smithi136 ceph-mon[38106]: pgmap v501: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:29:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:29:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:29:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:29:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:10 smithi136 ceph-mon[29560]: pgmap v501: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:29:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:29:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:29:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:29:10.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:10 smithi164 ceph-mon[33554]: pgmap v501: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:29:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:29:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:29:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: pgmap v504: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi136-74781-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: pgmap v504: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T05:29:11.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:29:11.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi136-74781-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:11.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:29:11.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:29:11.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T05:29:11.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: pgmap v504: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi136-74781-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T05:29:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:29:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:29:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:29:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:29:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[38106]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T05:29:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:29:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:29:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:29:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:29:12.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[29560]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T05:29:12.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:29:12.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:29:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:29:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:12 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:29:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:29:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:29:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:12 smithi164 ceph-mon[33554]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T05:29:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: pgmap v507: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:29:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: pgmap v507: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:29:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:29:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:29:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T05:29:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: pgmap v507: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:29:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:29:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:14 smithi136 ceph-mon[38106]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T05:29:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:15.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:29:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:14 smithi136 ceph-mon[29560]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T05:29:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:29:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:14 smithi164 ceph-mon[33554]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T05:29:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]: dispatch 2024-03-26T05:29:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: pgmap v510: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:29:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:29:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: pgmap v510: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:16.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T05:29:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: pgmap v510: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/499627439' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi136-74781-65"}]': finished 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:29:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:16 smithi136 ceph-mon[38106]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T05:29:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:17.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:16 smithi136 ceph-mon[29560]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T05:29:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:17.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:17.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi136-74781-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:29:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:16 smithi164 ceph-mon[33554]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T05:29:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:29:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:29:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:29:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:17 smithi136 ceph-mon[38106]: pgmap v513: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:29:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:17 smithi136 ceph-mon[38106]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T05:29:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:17 smithi136 ceph-mon[29560]: pgmap v513: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:29:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:17 smithi136 ceph-mon[29560]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T05:29:18.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:17 smithi164 ceph-mon[33554]: pgmap v513: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:29:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:17 smithi164 ceph-mon[33554]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T05:29:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:18 smithi164 ceph-mon[33554]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T05:29:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:18 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:18 smithi136 ceph-mon[38106]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T05:29:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:18 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi136-74781-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:18 smithi136 ceph-mon[29560]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T05:29:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:18 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:19 smithi164 ceph-mon[33554]: pgmap v516: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:29:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:19 smithi164 ceph-mon[33554]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T05:29:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:19 smithi136 ceph-mon[38106]: pgmap v516: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:29:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:19 smithi136 ceph-mon[38106]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T05:29:20.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:19 smithi136 ceph-mon[29560]: pgmap v516: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:29:20.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:19 smithi136 ceph-mon[29560]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T05:29:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:20 smithi164 ceph-mon[33554]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T05:29:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:20 smithi164 ceph-mon[33554]: pgmap v519: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:21.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:20 smithi136 ceph-mon[38106]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T05:29:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:20 smithi136 ceph-mon[38106]: pgmap v519: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:21.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:20 smithi136 ceph-mon[29560]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T05:29:21.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:21.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:21.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:20 smithi136 ceph-mon[29560]: pgmap v519: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3006 ms) 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7021 ms) 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7021 ms) 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7041 ms) 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7041 ms) 2024-03-26T05:29:21.869 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-26T05:29:21.870 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7020 ms) 2024-03-26T05:29:21.870 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-26T05:29:21.870 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7050 ms) 2024-03-26T05:29:21.870 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139358 ms total) 2024-03-26T05:29:21.870 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: 2024-03-26T05:29:21.870 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-26T05:29:21.870 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (310112 ms total) 2024-03-26T05:29:21.870 INFO:tasks.workunit.client.0.smithi136.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-26T05:29:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:21 smithi164 ceph-mon[33554]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T05:29:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:21 smithi136 ceph-mon[38106]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T05:29:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:21 smithi136 ceph-mon[29560]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T05:29:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/379985904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]: dispatch 2024-03-26T05:29:22.718 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:29:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:29:23.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:23.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:22 smithi164 ceph-mon[33554]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T05:29:23.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:22 smithi164 ceph-mon[33554]: pgmap v522: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:23.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:22 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:22 smithi136 ceph-mon[38106]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T05:29:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:22 smithi136 ceph-mon[38106]: pgmap v522: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:22 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:23.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi136-74781-66"}]': finished 2024-03-26T05:29:23.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:22 smithi136 ceph-mon[29560]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T05:29:23.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:22 smithi136 ceph-mon[29560]: pgmap v522: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:23.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:22 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:24.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:29:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:24 smithi164 ceph-mon[33554]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T05:29:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:24 smithi164 ceph-mon[33554]: pgmap v524: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:25.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[38106]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[38106]: pgmap v524: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[29560]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:24 smithi136 ceph-mon[29560]: pgmap v524: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:29:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]': finished 2024-03-26T05:29:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:25 smithi164 ceph-mon[33554]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T05:29:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]': finished 2024-03-26T05:29:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[38106]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T05:29:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]': finished 2024-03-26T05:29:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[29560]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T05:29:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-51"}]: dispatch 2024-03-26T05:29:27.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:26 smithi164 ceph-mon[33554]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T05:29:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:26 smithi164 ceph-mon[33554]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 250 B/s rd, 0 op/s 2024-03-26T05:29:27.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:26 smithi136 ceph-mon[38106]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T05:29:27.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:26 smithi136 ceph-mon[38106]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 250 B/s rd, 0 op/s 2024-03-26T05:29:27.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:26 smithi136 ceph-mon[29560]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T05:29:27.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:26 smithi136 ceph-mon[29560]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 250 B/s rd, 0 op/s 2024-03-26T05:29:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:29:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:29:28.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:27 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:29:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:27 smithi164 ceph-mon[33554]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T05:29:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[38106]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[29560]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:28 smithi164 ceph-mon[33554]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T05:29:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:28 smithi164 ceph-mon[33554]: pgmap v530: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:29:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-53"}]: dispatch 2024-03-26T05:29:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-53"}]: dispatch 2024-03-26T05:29:29.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[38106]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[38106]: pgmap v530: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-53"}]: dispatch 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-53"}]: dispatch 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[29560]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[29560]: pgmap v530: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:29:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-53"}]: dispatch 2024-03-26T05:29:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-53"}]: dispatch 2024-03-26T05:29:30.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:29 smithi164 ceph-mon[33554]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T05:29:30.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:29 smithi136 ceph-mon[38106]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T05:29:30.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:29 smithi136 ceph-mon[29560]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T05:29:31.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:30 smithi164 ceph-mon[33554]: pgmap v533: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:31.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:30 smithi164 ceph-mon[33554]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T05:29:31.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:31.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:31.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:30 smithi136 ceph-mon[38106]: pgmap v533: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:31.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:30 smithi136 ceph-mon[38106]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T05:29:31.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:31.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:30 smithi136 ceph-mon[29560]: pgmap v533: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:30 smithi136 ceph-mon[29560]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T05:29:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:31 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:31 smithi164 ceph-mon[33554]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T05:29:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-55"}]: dispatch 2024-03-26T05:29:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-55"}]: dispatch 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[38106]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-55"}]: dispatch 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-55"}]: dispatch 2024-03-26T05:29:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[29560]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T05:29:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-55"}]: dispatch 2024-03-26T05:29:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-55"}]: dispatch 2024-03-26T05:29:32.718 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:29:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:29:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:32 smithi164 ceph-mon[33554]: pgmap v535: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-26T05:29:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:32 smithi164 ceph-mon[33554]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T05:29:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:32 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:33.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:32 smithi136 ceph-mon[38106]: pgmap v535: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-26T05:29:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:32 smithi136 ceph-mon[38106]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T05:29:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:32 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:32 smithi136 ceph-mon[29560]: pgmap v535: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 507 B/s wr, 1 op/s 2024-03-26T05:29:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:32 smithi136 ceph-mon[29560]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T05:29:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:32 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:34.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:33 smithi164 ceph-mon[33554]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T05:29:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:34.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:33 smithi136 ceph-mon[38106]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T05:29:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:33 smithi136 ceph-mon[29560]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T05:29:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[38106]: pgmap v538: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[38106]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-57"}]: dispatch 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-57"}]: dispatch 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[29560]: pgmap v538: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[29560]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T05:29:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-57"}]: dispatch 2024-03-26T05:29:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-57"}]: dispatch 2024-03-26T05:29:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:34 smithi164 ceph-mon[33554]: pgmap v538: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:29:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:34 smithi164 ceph-mon[33554]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T05:29:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-57"}]: dispatch 2024-03-26T05:29:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-57"}]: dispatch 2024-03-26T05:29:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:35 smithi136 ceph-mon[29560]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T05:29:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:35 smithi136 ceph-mon[38106]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T05:29:36.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:35 smithi164 ceph-mon[33554]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[29560]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[29560]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[38106]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[38106]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T05:29:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:36 smithi164 ceph-mon[33554]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:29:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:36 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:36 smithi164 ceph-mon[33554]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T05:29:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:37.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:29:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:29:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:29:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[38106]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T05:29:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-59"}]: dispatch 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-59"}]: dispatch 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[29560]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-59"}]: dispatch 2024-03-26T05:29:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-59"}]: dispatch 2024-03-26T05:29:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:38 smithi164 ceph-mon[33554]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T05:29:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-59"}]: dispatch 2024-03-26T05:29:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-59"}]: dispatch 2024-03-26T05:29:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:39 smithi136 ceph-mon[38106]: pgmap v544: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:29:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:39 smithi136 ceph-mon[38106]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T05:29:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:39 smithi136 ceph-mon[29560]: pgmap v544: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:29:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:39 smithi136 ceph-mon[29560]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T05:29:39.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:39 smithi164 ceph-mon[33554]: pgmap v544: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:29:39.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:39 smithi164 ceph-mon[33554]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T05:29:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:40 smithi136 ceph-mon[38106]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T05:29:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:40 smithi136 ceph-mon[29560]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T05:29:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:40.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:40 smithi164 ceph-mon[33554]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T05:29:40.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:40.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[38106]: pgmap v547: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[38106]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T05:29:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:41.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[29560]: pgmap v547: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:41.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:41.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[29560]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T05:29:41.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:41.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:41.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:41 smithi164 ceph-mon[33554]: pgmap v547: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:41 smithi164 ceph-mon[33554]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T05:29:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[38106]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-61"}]: dispatch 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-61"}]: dispatch 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[29560]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T05:29:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:42.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:42.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-61"}]: dispatch 2024-03-26T05:29:42.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-61"}]: dispatch 2024-03-26T05:29:42.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:29:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:29:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:29:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:42 smithi164 ceph-mon[33554]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T05:29:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:29:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-61"}]: dispatch 2024-03-26T05:29:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-61"}]: dispatch 2024-03-26T05:29:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[38106]: pgmap v550: 260 pgs: 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:29:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[38106]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T05:29:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[29560]: pgmap v550: 260 pgs: 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:29:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[29560]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T05:29:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:43 smithi164 ceph-mon[33554]: pgmap v550: 260 pgs: 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:29:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:43 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:29:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:43 smithi164 ceph-mon[33554]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T05:29:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:44 smithi136 ceph-mon[38106]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T05:29:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:44 smithi136 ceph-mon[29560]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T05:29:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:44 smithi164 ceph-mon[33554]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T05:29:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:29:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:29:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[38106]: pgmap v553: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:29:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[38106]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T05:29:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:29:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[38106]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T05:29:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[29560]: pgmap v553: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:29:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[29560]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T05:29:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:29:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:45 smithi136 ceph-mon[29560]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T05:29:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:45 smithi164 ceph-mon[33554]: pgmap v553: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:29:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:45 smithi164 ceph-mon[33554]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T05:29:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:29:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:29:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:29:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:45 smithi164 ceph-mon[33554]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T05:29:47.326 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:29:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:29:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:29:47.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:47 smithi136 ceph-mon[38106]: pgmap v556: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T05:29:47.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:47 smithi136 ceph-mon[38106]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T05:29:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:47 smithi136 ceph-mon[29560]: pgmap v556: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T05:29:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:47 smithi136 ceph-mon[29560]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T05:29:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:47 smithi164 ceph-mon[33554]: pgmap v556: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T05:29:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:47 smithi164 ceph-mon[33554]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T05:29:48.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:48 smithi164 ceph-mon[33554]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T05:29:48.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:48 smithi136 ceph-mon[38106]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T05:29:48.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:48 smithi136 ceph-mon[29560]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T05:29:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:49 smithi164 ceph-mon[33554]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T05:29:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:49 smithi164 ceph-mon[33554]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T05:29:49.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:49 smithi136 ceph-mon[38106]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T05:29:49.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:49 smithi136 ceph-mon[38106]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T05:29:49.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:49 smithi136 ceph-mon[29560]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T05:29:49.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:49 smithi136 ceph-mon[29560]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T05:29:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:50 smithi164 ceph-mon[33554]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T05:29:50.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:50 smithi136 ceph-mon[38106]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T05:29:50.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:50 smithi136 ceph-mon[29560]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T05:29:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:51 smithi164 ceph-mon[33554]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:51.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:51 smithi136 ceph-mon[38106]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:51.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:51 smithi136 ceph-mon[29560]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:29:52.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:29:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:29:52.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:52 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:29:52.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:52 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:29:52.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:52 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:29:52.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:52 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:29:52.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:52 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:29:52.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:52 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:29:52.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:52 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:29:52.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:52 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:29:52.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:52 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:29:52.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:52 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:29:52.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:52 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:29:52.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:52 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:29:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:53 smithi164 ceph-mon[33554]: pgmap v563: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-26T05:29:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:53 smithi136 ceph-mon[38106]: pgmap v563: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-26T05:29:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:53.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:53 smithi136 ceph-mon[29560]: pgmap v563: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-26T05:29:53.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:29:54.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:54 smithi136 ceph-mon[38106]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T05:29:54.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:54 smithi136 ceph-mon[29560]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T05:29:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:54 smithi164 ceph-mon[33554]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T05:29:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:55 smithi136 ceph-mon[38106]: pgmap v565: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:29:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:55 smithi136 ceph-mon[29560]: pgmap v565: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:29:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:55 smithi164 ceph-mon[33554]: pgmap v565: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:29:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:29:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:29:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:29:57.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:57 smithi136 ceph-mon[38106]: pgmap v566: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 780 B/s wr, 3 op/s 2024-03-26T05:29:57.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:57 smithi136 ceph-mon[29560]: pgmap v566: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 780 B/s wr, 3 op/s 2024-03-26T05:29:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:57 smithi164 ceph-mon[33554]: pgmap v566: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 780 B/s wr, 3 op/s 2024-03-26T05:29:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:29:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:29:59 smithi136 ceph-mon[38106]: pgmap v567: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 693 B/s wr, 3 op/s 2024-03-26T05:29:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:29:59 smithi136 ceph-mon[29560]: pgmap v567: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 693 B/s wr, 3 op/s 2024-03-26T05:29:59.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:29:59 smithi164 ceph-mon[33554]: pgmap v567: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 693 B/s wr, 3 op/s 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[38106]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[38106]: Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[38106]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[38106]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[38106]: application not enabled on pool 'SetOpFlagssmithi136-76022-1' 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[38106]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[29560]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[29560]: Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:30:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[29560]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-26T05:30:00.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[29560]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T05:30:00.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[29560]: application not enabled on pool 'SetOpFlagssmithi136-76022-1' 2024-03-26T05:30:00.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:00 smithi136 ceph-mon[29560]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T05:30:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:00 smithi164 ceph-mon[33554]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T05:30:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:00 smithi164 ceph-mon[33554]: Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:30:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:00 smithi164 ceph-mon[33554]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-26T05:30:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:00 smithi164 ceph-mon[33554]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T05:30:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:00 smithi164 ceph-mon[33554]: application not enabled on pool 'SetOpFlagssmithi136-76022-1' 2024-03-26T05:30:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:00 smithi164 ceph-mon[33554]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T05:30:01.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:01 smithi136 ceph-mon[29560]: pgmap v569: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:30:01.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:01 smithi136 ceph-mon[38106]: pgmap v569: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:30:01.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:01 smithi164 ceph-mon[33554]: pgmap v569: 260 pgs: 260 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:30:02.718 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:30:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:30:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:03 smithi136 ceph-mon[29560]: pgmap v570: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:03 smithi136 ceph-mon[29560]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T05:30:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:03 smithi136 ceph-mon[38106]: pgmap v570: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:03 smithi136 ceph-mon[38106]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T05:30:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:03 smithi164 ceph-mon[33554]: pgmap v570: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:03 smithi164 ceph-mon[33554]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T05:30:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:05 smithi136 ceph-mon[38106]: pgmap v572: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:05 smithi136 ceph-mon[29560]: pgmap v572: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:05 smithi164 ceph-mon[33554]: pgmap v572: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:07.504 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:30:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:30:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:30:07.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:07 smithi136 ceph-mon[38106]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:07 smithi136 ceph-mon[29560]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:07.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:07 smithi164 ceph-mon[33554]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:09.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:09 smithi164 ceph-mon[33554]: pgmap v574: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:09.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:09 smithi136 ceph-mon[38106]: pgmap v574: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:09 smithi136 ceph-mon[29560]: pgmap v574: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:10.790 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:10 smithi164 ceph-mon[33554]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T05:30:10.790 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]: dispatch 2024-03-26T05:30:10.791 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]: dispatch 2024-03-26T05:30:10.791 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:10 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]: dispatch 2024-03-26T05:30:10.791 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:10 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]: dispatch 2024-03-26T05:30:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[38106]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T05:30:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]: dispatch 2024-03-26T05:30:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]: dispatch 2024-03-26T05:30:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]: dispatch 2024-03-26T05:30:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]: dispatch 2024-03-26T05:30:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[29560]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T05:30:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]: dispatch 2024-03-26T05:30:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]: dispatch 2024-03-26T05:30:10.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]: dispatch 2024-03-26T05:30:10.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:10 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]: dispatch 2024-03-26T05:30:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:11 smithi136 ceph-mon[29560]: pgmap v576: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:30:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]': finished 2024-03-26T05:30:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]': finished 2024-03-26T05:30:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:11 smithi136 ceph-mon[29560]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T05:30:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:11 smithi136 ceph-mon[38106]: pgmap v576: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:30:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]': finished 2024-03-26T05:30:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]': finished 2024-03-26T05:30:11.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:11 smithi136 ceph-mon[38106]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T05:30:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:11 smithi164 ceph-mon[33554]: pgmap v576: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:30:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.1", "id": [5, 1]}]': finished 2024-03-26T05:30:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.11", "id": [7, 4]}]': finished 2024-03-26T05:30:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:11 smithi164 ceph-mon[33554]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T05:30:12.538 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:30:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:30:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:12 smithi136 ceph-mon[38106]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T05:30:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:12 smithi136 ceph-mon[29560]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T05:30:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:12 smithi164 ceph-mon[33554]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T05:30:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:13 smithi136 ceph-mon[38106]: pgmap v579: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:30:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:13 smithi136 ceph-mon[38106]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T05:30:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:13 smithi136 ceph-mon[29560]: pgmap v579: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:30:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:13 smithi136 ceph-mon[29560]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T05:30:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:13 smithi164 ceph-mon[33554]: pgmap v579: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:30:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:13 smithi164 ceph-mon[33554]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T05:30:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:15 smithi136 ceph-mon[38106]: pgmap v581: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:30:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:15 smithi136 ceph-mon[29560]: pgmap v581: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:30:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:15 smithi164 ceph-mon[33554]: pgmap v581: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:30:17.544 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:30:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:30:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:30:17.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:17 smithi136 ceph-mon[38106]: pgmap v582: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-26T05:30:17.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:17 smithi136 ceph-mon[29560]: pgmap v582: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-26T05:30:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:17 smithi164 ceph-mon[33554]: pgmap v582: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-26T05:30:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:19 smithi136 ceph-mon[38106]: pgmap v583: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:30:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:19 smithi136 ceph-mon[29560]: pgmap v583: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:30:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:19 smithi164 ceph-mon[33554]: pgmap v583: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T05:30:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:20.824 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-63"}]: dispatch 2024-03-26T05:30:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-63"}]: dispatch 2024-03-26T05:30:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-63"}]: dispatch 2024-03-26T05:30:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-63"}]: dispatch 2024-03-26T05:30:20.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-63"}]: dispatch 2024-03-26T05:30:20.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-63"}]: dispatch 2024-03-26T05:30:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:21 smithi136 ceph-mon[38106]: pgmap v584: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-03-26T05:30:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:21 smithi136 ceph-mon[38106]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T05:30:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:21 smithi136 ceph-mon[29560]: pgmap v584: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-03-26T05:30:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:21 smithi136 ceph-mon[29560]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T05:30:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:21 smithi164 ceph-mon[33554]: pgmap v584: 260 pgs: 260 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-03-26T05:30:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:21 smithi164 ceph-mon[33554]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T05:30:22.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:30:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:30:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:22 smithi136 ceph-mon[38106]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T05:30:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:22.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:22.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:22 smithi136 ceph-mon[29560]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T05:30:22.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:22.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:22 smithi164 ceph-mon[33554]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T05:30:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[38106]: pgmap v587: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[38106]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[29560]: pgmap v587: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[29560]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T05:30:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:23 smithi164 ceph-mon[33554]: pgmap v587: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:23 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:23 smithi164 ceph-mon[33554]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T05:30:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:30:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:24 smithi164 ceph-mon[33554]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T05:30:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:30:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:24 smithi136 ceph-mon[29560]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T05:30:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:30:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:24 smithi136 ceph-mon[38106]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T05:30:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:25 smithi164 ceph-mon[33554]: pgmap v590: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:25 smithi164 ceph-mon[33554]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T05:30:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:25 smithi136 ceph-mon[29560]: pgmap v590: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:25 smithi136 ceph-mon[29560]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T05:30:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:25 smithi136 ceph-mon[38106]: pgmap v590: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:25 smithi136 ceph-mon[38106]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T05:30:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:26 smithi164 ceph-mon[33554]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T05:30:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:26 smithi136 ceph-mon[38106]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T05:30:27.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:26 smithi136 ceph-mon[29560]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T05:30:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:30:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:30:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:30:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:27 smithi164 ceph-mon[33554]: pgmap v593: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-26T05:30:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:27 smithi164 ceph-mon[33554]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T05:30:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:27 smithi136 ceph-mon[38106]: pgmap v593: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-26T05:30:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:27 smithi136 ceph-mon[38106]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T05:30:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:27 smithi136 ceph-mon[29560]: pgmap v593: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-26T05:30:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:27 smithi136 ceph-mon[29560]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T05:30:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:28 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:28 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:28 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:29 smithi164 ceph-mon[33554]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-26T05:30:30.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:29 smithi136 ceph-mon[29560]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-26T05:30:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:29 smithi136 ceph-mon[38106]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-26T05:30:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:31 smithi164 ceph-mon[33554]: pgmap v596: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T05:30:32.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:31 smithi136 ceph-mon[38106]: pgmap v596: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T05:30:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:31 smithi136 ceph-mon[29560]: pgmap v596: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T05:30:32.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:30:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:30:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:33 smithi164 ceph-mon[33554]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-26T05:30:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:33 smithi164 ceph-mon[33554]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T05:30:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:33 smithi136 ceph-mon[38106]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-26T05:30:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:33 smithi136 ceph-mon[38106]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T05:30:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:33 smithi136 ceph-mon[29560]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-26T05:30:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:33 smithi136 ceph-mon[29560]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T05:30:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:35 smithi164 ceph-mon[33554]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:35 smithi136 ceph-mon[38106]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:35 smithi136 ceph-mon[29560]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:30:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:30:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:37 smithi164 ceph-mon[33554]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-65"}]: dispatch 2024-03-26T05:30:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-65"}]: dispatch 2024-03-26T05:30:38.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[38106]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-65"}]: dispatch 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-65"}]: dispatch 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[29560]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-65"}]: dispatch 2024-03-26T05:30:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-65"}]: dispatch 2024-03-26T05:30:38.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:38 smithi164 ceph-mon[33554]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T05:30:39.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:38 smithi136 ceph-mon[38106]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T05:30:39.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:38 smithi136 ceph-mon[29560]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T05:30:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:39 smithi164 ceph-mon[33554]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:39 smithi164 ceph-mon[33554]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T05:30:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:39 smithi136 ceph-mon[38106]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:39 smithi136 ceph-mon[38106]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T05:30:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:39 smithi136 ceph-mon[29560]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:39 smithi136 ceph-mon[29560]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T05:30:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:40.968 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:40 smithi164 ceph-mon[33554]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T05:30:40.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:40.975 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:40.975 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:40 smithi136 ceph-mon[29560]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T05:30:40.975 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:40.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:40.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:40 smithi136 ceph-mon[38106]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T05:30:40.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:41 smithi164 ceph-mon[33554]: pgmap v605: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:41 smithi164 ceph-mon[33554]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T05:30:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-67"}]: dispatch 2024-03-26T05:30:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-67"}]: dispatch 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[38106]: pgmap v605: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[38106]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-67"}]: dispatch 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-67"}]: dispatch 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[29560]: pgmap v605: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[29560]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T05:30:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:42.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:42.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-67"}]: dispatch 2024-03-26T05:30:42.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-67"}]: dispatch 2024-03-26T05:30:42.665 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:30:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:30:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:42 smithi164 ceph-mon[33554]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T05:30:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:42 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:43.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:42 smithi136 ceph-mon[38106]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T05:30:43.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:42 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:42 smithi136 ceph-mon[29560]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T05:30:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:42 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:43 smithi164 ceph-mon[33554]: pgmap v608: 228 pgs: 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:30:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:43 smithi164 ceph-mon[33554]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T05:30:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[38106]: pgmap v608: 228 pgs: 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[38106]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[29560]: pgmap v608: 228 pgs: 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[29560]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:44 smithi164 ceph-mon[33554]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T05:30:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-69"}]: dispatch 2024-03-26T05:30:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-69"}]: dispatch 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[38106]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-69"}]: dispatch 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-69"}]: dispatch 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[29560]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:45.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:30:45.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-69"}]: dispatch 2024-03-26T05:30:45.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-69"}]: dispatch 2024-03-26T05:30:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:45 smithi164 ceph-mon[33554]: pgmap v611: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:30:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:45 smithi164 ceph-mon[33554]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T05:30:46.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:45 smithi136 ceph-mon[38106]: pgmap v611: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:30:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:45 smithi136 ceph-mon[38106]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T05:30:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:45 smithi136 ceph-mon[29560]: pgmap v611: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:30:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:45 smithi136 ceph-mon[29560]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T05:30:46.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:46 smithi164 ceph-mon[33554]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T05:30:46.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:46.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:47.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:46 smithi136 ceph-mon[38106]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T05:30:47.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:47.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:47.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:46 smithi136 ceph-mon[29560]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T05:30:47.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:47.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:30:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:30:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:30:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:47 smithi164 ceph-mon[33554]: pgmap v614: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:30:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:47 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:47 smithi164 ceph-mon[33554]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T05:30:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:48.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[38106]: pgmap v614: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[38106]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[29560]: pgmap v614: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[29560]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T05:30:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:30:48.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:48.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:30:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:30:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:48 smithi164 ceph-mon[33554]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T05:30:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:30:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:48 smithi136 ceph-mon[38106]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T05:30:49.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:30:49.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:48 smithi136 ceph-mon[29560]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T05:30:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:49 smithi164 ceph-mon[33554]: pgmap v617: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:30:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:49 smithi164 ceph-mon[33554]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T05:30:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:49 smithi136 ceph-mon[38106]: pgmap v617: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:30:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:49 smithi136 ceph-mon[38106]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T05:30:50.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:49 smithi136 ceph-mon[29560]: pgmap v617: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:30:50.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:49 smithi136 ceph-mon[29560]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T05:30:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:51 smithi136 ceph-mon[38106]: pgmap v619: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 717 B/s wr, 2 op/s 2024-03-26T05:30:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:51 smithi136 ceph-mon[29560]: pgmap v619: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 717 B/s wr, 2 op/s 2024-03-26T05:30:52.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:51 smithi164 ceph-mon[33554]: pgmap v619: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 717 B/s wr, 2 op/s 2024-03-26T05:30:52.718 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:30:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:30:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:52 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:30:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:52 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:30:53.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:52 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:30:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[29560]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T05:30:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:30:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:30:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:30:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[38106]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T05:30:54.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:54.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:30:54.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:30:54.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:30:54.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:53 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:53 smithi164 ceph-mon[33554]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T05:30:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:30:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:53 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:30:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:53 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:30:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:53 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:30:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:53 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:30:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:55 smithi136 ceph-mon[38106]: pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T05:30:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:55 smithi136 ceph-mon[29560]: pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T05:30:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:55 smithi164 ceph-mon[33554]: pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T05:30:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:30:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:30:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:30:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:57 smithi136 ceph-mon[38106]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T05:30:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:57 smithi136 ceph-mon[29560]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T05:30:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:30:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:57 smithi164 ceph-mon[33554]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T05:30:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:30:59 smithi136 ceph-mon[38106]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T05:31:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:30:59 smithi136 ceph-mon[29560]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T05:31:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:30:59 smithi164 ceph-mon[33554]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T05:31:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:01 smithi136 ceph-mon[38106]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T05:31:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:01 smithi136 ceph-mon[29560]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T05:31:02.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:01 smithi164 ceph-mon[33554]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T05:31:02.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:31:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:31:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:03 smithi136 ceph-mon[38106]: pgmap v625: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T05:31:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:03 smithi136 ceph-mon[29560]: pgmap v625: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T05:31:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:03 smithi164 ceph-mon[33554]: pgmap v625: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T05:31:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:05 smithi136 ceph-mon[38106]: pgmap v626: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:05 smithi136 ceph-mon[29560]: pgmap v626: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:05 smithi164 ceph-mon[33554]: pgmap v626: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:31:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:31:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:31:08.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:07 smithi136 ceph-mon[29560]: pgmap v627: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:07 smithi136 ceph-mon[38106]: pgmap v627: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:07 smithi164 ceph-mon[33554]: pgmap v627: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:10.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:09 smithi164 ceph-mon[33554]: pgmap v628: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:10.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:10.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:10.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-71"}]: dispatch 2024-03-26T05:31:10.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-71"}]: dispatch 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[38106]: pgmap v628: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-71"}]: dispatch 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-71"}]: dispatch 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[29560]: pgmap v628: 260 pgs: 260 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-71"}]: dispatch 2024-03-26T05:31:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-71"}]: dispatch 2024-03-26T05:31:11.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:10 smithi136 ceph-mon[38106]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T05:31:11.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:10 smithi136 ceph-mon[29560]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T05:31:11.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:10 smithi164 ceph-mon[33554]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T05:31:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:11 smithi136 ceph-mon[38106]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T05:31:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:11 smithi136 ceph-mon[38106]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T05:31:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:11 smithi136 ceph-mon[29560]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T05:31:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:11 smithi136 ceph-mon[29560]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T05:31:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:11 smithi164 ceph-mon[33554]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T05:31:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:11 smithi164 ceph-mon[33554]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T05:31:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:12.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:31:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:31:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[38106]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T05:31:13.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:13.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:13.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[38106]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T05:31:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[29560]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T05:31:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:12 smithi136 ceph-mon[29560]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T05:31:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:12 smithi164 ceph-mon[33554]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T05:31:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:12 smithi164 ceph-mon[33554]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T05:31:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:13 smithi136 ceph-mon[38106]: pgmap v633: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:13 smithi136 ceph-mon[38106]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T05:31:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:13 smithi136 ceph-mon[29560]: pgmap v633: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:13 smithi136 ceph-mon[29560]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T05:31:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:13 smithi164 ceph-mon[33554]: pgmap v633: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:13 smithi164 ceph-mon[33554]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T05:31:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:15 smithi136 ceph-mon[38106]: pgmap v636: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:31:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:15 smithi136 ceph-mon[29560]: pgmap v636: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:31:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:15 smithi164 ceph-mon[33554]: pgmap v636: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:31:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:31:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:31:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:31:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:17 smithi136 ceph-mon[38106]: pgmap v637: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-26T05:31:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:17 smithi136 ceph-mon[29560]: pgmap v637: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-26T05:31:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:17 smithi164 ceph-mon[33554]: pgmap v637: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-26T05:31:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:19 smithi136 ceph-mon[38106]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T05:31:20.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:19 smithi136 ceph-mon[29560]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T05:31:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:19 smithi164 ceph-mon[33554]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T05:31:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:21 smithi136 ceph-mon[38106]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T05:31:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:21 smithi136 ceph-mon[29560]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T05:31:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:21 smithi164 ceph-mon[33554]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T05:31:22.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:31:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:31:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:23 smithi136 ceph-mon[38106]: pgmap v640: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 777 B/s wr, 3 op/s 2024-03-26T05:31:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:23 smithi136 ceph-mon[29560]: pgmap v640: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 777 B/s wr, 3 op/s 2024-03-26T05:31:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:23 smithi164 ceph-mon[33554]: pgmap v640: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 777 B/s wr, 3 op/s 2024-03-26T05:31:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-73"}]: dispatch 2024-03-26T05:31:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-73"}]: dispatch 2024-03-26T05:31:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-73"}]: dispatch 2024-03-26T05:31:25.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-73"}]: dispatch 2024-03-26T05:31:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-73"}]: dispatch 2024-03-26T05:31:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-73"}]: dispatch 2024-03-26T05:31:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:25 smithi136 ceph-mon[38106]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 701 B/s wr, 3 op/s 2024-03-26T05:31:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:25 smithi136 ceph-mon[38106]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T05:31:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:25 smithi136 ceph-mon[29560]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 701 B/s wr, 3 op/s 2024-03-26T05:31:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:25 smithi136 ceph-mon[29560]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T05:31:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:25 smithi164 ceph-mon[33554]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 701 B/s wr, 3 op/s 2024-03-26T05:31:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:25 smithi164 ceph-mon[33554]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T05:31:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:26 smithi164 ceph-mon[33554]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T05:31:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:27.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:26 smithi136 ceph-mon[38106]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T05:31:27.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:27.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:27.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:26 smithi136 ceph-mon[29560]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T05:31:27.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:27.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:31:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:31:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:27 smithi164 ceph-mon[33554]: pgmap v644: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:31:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:27 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:31:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:27 smithi164 ceph-mon[33554]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T05:31:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[38106]: pgmap v644: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:31:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:31:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[38106]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T05:31:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[29560]: pgmap v644: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:31:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:31:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[29560]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T05:31:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:28 smithi164 ceph-mon[33554]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T05:31:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:28 smithi136 ceph-mon[38106]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T05:31:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:28 smithi136 ceph-mon[29560]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T05:31:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:29 smithi164 ceph-mon[33554]: pgmap v647: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:29 smithi164 ceph-mon[33554]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T05:31:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-75"}]: dispatch 2024-03-26T05:31:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-75"}]: dispatch 2024-03-26T05:31:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[38106]: pgmap v647: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[38106]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T05:31:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-75"}]: dispatch 2024-03-26T05:31:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-75"}]: dispatch 2024-03-26T05:31:30.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[29560]: pgmap v647: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[29560]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T05:31:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-75"}]: dispatch 2024-03-26T05:31:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-75"}]: dispatch 2024-03-26T05:31:31.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:30 smithi164 ceph-mon[33554]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T05:31:31.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:30 smithi136 ceph-mon[38106]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T05:31:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:30 smithi136 ceph-mon[29560]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T05:31:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:31 smithi164 ceph-mon[33554]: pgmap v650: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:31 smithi164 ceph-mon[33554]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T05:31:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:31 smithi136 ceph-mon[38106]: pgmap v650: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:31 smithi136 ceph-mon[38106]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T05:31:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:31 smithi136 ceph-mon[29560]: pgmap v650: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:31:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:31 smithi136 ceph-mon[29560]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T05:31:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:32.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:31:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:31:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:32 smithi164 ceph-mon[33554]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T05:31:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:32 smithi136 ceph-mon[38106]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T05:31:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:32 smithi136 ceph-mon[29560]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T05:31:33.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:33 smithi164 ceph-mon[33554]: pgmap v653: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:33 smithi164 ceph-mon[33554]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T05:31:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-77"}]: dispatch 2024-03-26T05:31:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-77"}]: dispatch 2024-03-26T05:31:34.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:33 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:31:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[38106]: pgmap v653: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[38106]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T05:31:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-77"}]: dispatch 2024-03-26T05:31:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-77"}]: dispatch 2024-03-26T05:31:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:31:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[29560]: pgmap v653: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[29560]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T05:31:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-77"}]: dispatch 2024-03-26T05:31:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-77"}]: dispatch 2024-03-26T05:31:34.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:33 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:31:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:34 smithi164 ceph-mon[33554]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T05:31:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:34 smithi164 ceph-mon[33554]: pgmap v656: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:34 smithi136 ceph-mon[38106]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T05:31:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:34 smithi136 ceph-mon[38106]: pgmap v656: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:34 smithi136 ceph-mon[29560]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T05:31:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:34 smithi136 ceph-mon[29560]: pgmap v656: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:35 smithi164 ceph-mon[33554]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T05:31:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:35 smithi136 ceph-mon[29560]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T05:31:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:35 smithi136 ceph-mon[38106]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T05:31:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:31:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:36 smithi164 ceph-mon[33554]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T05:31:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:36 smithi164 ceph-mon[33554]: pgmap v659: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:36 smithi136 ceph-mon[29560]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T05:31:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:36 smithi136 ceph-mon[29560]: pgmap v659: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:31:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:36 smithi136 ceph-mon[38106]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T05:31:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:31:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:36 smithi136 ceph-mon[38106]: pgmap v659: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:31:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:31:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:31:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:37 smithi164 ceph-mon[33554]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T05:31:38.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:37 smithi136 ceph-mon[38106]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T05:31:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:37 smithi136 ceph-mon[29560]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T05:31:39.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:38 smithi164 ceph-mon[33554]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T05:31:39.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.14"}]: dispatch 2024-03-26T05:31:39.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:38 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.14"}]: dispatch 2024-03-26T05:31:39.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:38 smithi164 ceph-mon[33554]: pgmap v662: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:38 smithi136 ceph-mon[38106]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T05:31:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:38 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.14"}]: dispatch 2024-03-26T05:31:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:38 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.14"}]: dispatch 2024-03-26T05:31:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:38 smithi136 ceph-mon[38106]: pgmap v662: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:38 smithi136 ceph-mon[29560]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T05:31:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.14"}]: dispatch 2024-03-26T05:31:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:38 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.14"}]: dispatch 2024-03-26T05:31:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:38 smithi136 ceph-mon[29560]: pgmap v662: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:31:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:39 smithi164 ceph-mon[33554]: 292.14 deep-scrub starts 2024-03-26T05:31:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:39 smithi164 ceph-mon[33554]: 292.14 deep-scrub ok 2024-03-26T05:31:40.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:39 smithi136 ceph-mon[38106]: 292.14 deep-scrub starts 2024-03-26T05:31:40.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:39 smithi136 ceph-mon[38106]: 292.14 deep-scrub ok 2024-03-26T05:31:40.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:39 smithi136 ceph-mon[29560]: 292.14 deep-scrub starts 2024-03-26T05:31:40.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:39 smithi136 ceph-mon[29560]: 292.14 deep-scrub ok 2024-03-26T05:31:41.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:40 smithi164 ceph-mon[33554]: pgmap v663: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 601 B/s wr, 2 op/s 2024-03-26T05:31:41.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:40 smithi136 ceph-mon[29560]: pgmap v663: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 601 B/s wr, 2 op/s 2024-03-26T05:31:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:40 smithi136 ceph-mon[38106]: pgmap v663: 260 pgs: 2 creating+activating, 18 creating+peering, 240 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 601 B/s wr, 2 op/s 2024-03-26T05:31:42.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:31:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:31:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:43 smithi136 ceph-mon[29560]: pgmap v664: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 838 B/s wr, 119 op/s 2024-03-26T05:31:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:43 smithi136 ceph-mon[38106]: pgmap v664: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 838 B/s wr, 119 op/s 2024-03-26T05:31:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:43 smithi164 ceph-mon[33554]: pgmap v664: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 838 B/s wr, 119 op/s 2024-03-26T05:31:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:45 smithi136 ceph-mon[38106]: pgmap v665: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 511 B/s wr, 91 op/s 2024-03-26T05:31:45.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:45 smithi136 ceph-mon[29560]: pgmap v665: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 511 B/s wr, 91 op/s 2024-03-26T05:31:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:45 smithi164 ceph-mon[33554]: pgmap v665: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 511 B/s wr, 91 op/s 2024-03-26T05:31:47.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:47 smithi136 ceph-mon[38106]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 450 B/s wr, 80 op/s 2024-03-26T05:31:47.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:47 smithi136 ceph-mon[29560]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 450 B/s wr, 80 op/s 2024-03-26T05:31:47.326 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:31:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:31:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:31:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:47 smithi164 ceph-mon[33554]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 450 B/s wr, 80 op/s 2024-03-26T05:31:49.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:49 smithi136 ceph-mon[38106]: pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 406 B/s wr, 72 op/s 2024-03-26T05:31:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:49 smithi136 ceph-mon[29560]: pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 406 B/s wr, 72 op/s 2024-03-26T05:31:49.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:49 smithi164 ceph-mon[33554]: pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 406 B/s wr, 72 op/s 2024-03-26T05:31:51.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:51 smithi136 ceph-mon[38106]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 341 B/s wr, 61 op/s 2024-03-26T05:31:51.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:51 smithi136 ceph-mon[29560]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 341 B/s wr, 61 op/s 2024-03-26T05:31:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:51 smithi164 ceph-mon[33554]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 341 B/s wr, 61 op/s 2024-03-26T05:31:52.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:31:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:31:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:53 smithi136 ceph-mon[38106]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 341 B/s wr, 61 op/s 2024-03-26T05:31:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:53 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:31:53.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:53 smithi136 ceph-mon[29560]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 341 B/s wr, 61 op/s 2024-03-26T05:31:53.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:53.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:53 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:31:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:53 smithi164 ceph-mon[33554]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 341 B/s wr, 61 op/s 2024-03-26T05:31:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:31:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:53 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:31:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:31:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:31:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:31:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:31:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:31:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:31:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:31:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:31:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:31:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:55 smithi164 ceph-mon[33554]: pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:31:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:55 smithi136 ceph-mon[38106]: pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:31:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:55 smithi136 ceph-mon[29560]: pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:31:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:31:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:31:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:31:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:57 smithi136 ceph-mon[29560]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:31:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:57 smithi136 ceph-mon[38106]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:31:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:57 smithi164 ceph-mon[33554]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:31:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:31:58.219 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:31:57 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:31:57.950+0000 7f2e6fa54700 -1 snap_mapper.add_oid found existing snaps mapped on 292:2b26c11a:test-rados-api-smithi136-75071-80::foo:2, removing 2024-03-26T05:31:58.325 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:31:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0[42137]: 2024-03-26T05:31:57.950+0000 7f01ce194700 -1 snap_mapper.add_oid found existing snaps mapped on 292:2b26c11a:test-rados-api-smithi136-75071-80::foo:2, removing 2024-03-26T05:31:58.325 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:31:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3[59564]: 2024-03-26T05:31:57.951+0000 7f64b1d24700 -1 snap_mapper.add_oid found existing snaps mapped on 292:2b26c11a:test-rados-api-smithi136-75071-80::foo:2, removing 2024-03-26T05:31:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-79"}]: dispatch 2024-03-26T05:31:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-79"}]: dispatch 2024-03-26T05:31:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-79"}]: dispatch 2024-03-26T05:31:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-79"}]: dispatch 2024-03-26T05:31:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:31:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-79"}]: dispatch 2024-03-26T05:31:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-79"}]: dispatch 2024-03-26T05:31:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:59 smithi164 ceph-mon[33554]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:31:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:31:59 smithi164 ceph-mon[33554]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T05:31:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:59 smithi136 ceph-mon[38106]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:31:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:31:59 smithi136 ceph-mon[38106]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T05:31:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:59 smithi136 ceph-mon[29560]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:31:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:31:59 smithi136 ceph-mon[29560]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T05:32:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:00 smithi136 ceph-mon[38106]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T05:32:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:00.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:00 smithi136 ceph-mon[29560]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T05:32:00.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:00.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:00 smithi164 ceph-mon[33554]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T05:32:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[38106]: pgmap v675: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:32:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[38106]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T05:32:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:01.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[29560]: pgmap v675: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:32:01.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:01.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:01.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[29560]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T05:32:01.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:01.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:01.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:01 smithi164 ceph-mon[33554]: pgmap v675: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:32:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:01 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:01 smithi164 ceph-mon[33554]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T05:32:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:02.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:32:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:32:02.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:02.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:02.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:02 smithi164 ceph-mon[33554]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T05:32:02.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:02 smithi136 ceph-mon[38106]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T05:32:02.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:02 smithi136 ceph-mon[29560]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T05:32:02.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[38106]: pgmap v678: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T05:32:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[38106]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T05:32:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:03.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[29560]: pgmap v678: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T05:32:03.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:03.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:03.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[29560]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T05:32:03.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:03.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:03 smithi164 ceph-mon[33554]: pgmap v678: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T05:32:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:03 smithi164 ceph-mon[33554]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T05:32:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[38106]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T05:32:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[29560]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T05:32:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:04.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:04.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:04 smithi164 ceph-mon[33554]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T05:32:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:05 smithi136 ceph-mon[38106]: pgmap v681: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:32:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:05 smithi136 ceph-mon[38106]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T05:32:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T05:32:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T05:32:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:05 smithi136 ceph-mon[29560]: pgmap v681: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:32:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:05 smithi136 ceph-mon[29560]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T05:32:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T05:32:05.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T05:32:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:05 smithi164 ceph-mon[33554]: pgmap v681: 260 pgs: 260 active+clean; 8.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:32:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:05 smithi164 ceph-mon[33554]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T05:32:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T05:32:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T05:32:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T05:32:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:06 smithi136 ceph-mon[38106]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T05:32:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T05:32:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:06 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T05:32:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T05:32:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:06 smithi136 ceph-mon[29560]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T05:32:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T05:32:06.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:06 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T05:32:06.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T05:32:06.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:06 smithi164 ceph-mon[33554]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T05:32:06.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T05:32:06.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:06 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T05:32:07.516 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:32:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:32:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[29560]: pgmap v684: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T05:32:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T05:32:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[29560]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T05:32:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:07.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[38106]: pgmap v684: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T05:32:07.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T05:32:07.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[38106]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T05:32:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:07 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:07 smithi164 ceph-mon[33554]: pgmap v684: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T05:32:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T05:32:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:07 smithi164 ceph-mon[33554]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T05:32:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:07 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[29560]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T05:32:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-81"}]: dispatch 2024-03-26T05:32:08.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-81"}]: dispatch 2024-03-26T05:32:08.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:08.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[38106]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T05:32:08.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:08.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:08.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-81"}]: dispatch 2024-03-26T05:32:08.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:08 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-81"}]: dispatch 2024-03-26T05:32:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:08 smithi164 ceph-mon[33554]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T05:32:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-81"}]: dispatch 2024-03-26T05:32:08.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:08 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-81"}]: dispatch 2024-03-26T05:32:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:09 smithi136 ceph-mon[38106]: pgmap v687: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T05:32:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:09 smithi136 ceph-mon[38106]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T05:32:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:09 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:09 smithi136 ceph-mon[29560]: pgmap v687: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T05:32:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:09 smithi136 ceph-mon[29560]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T05:32:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:09 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:09 smithi164 ceph-mon[33554]: pgmap v687: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T05:32:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:09 smithi164 ceph-mon[33554]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T05:32:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:09 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:10 smithi136 ceph-mon[38106]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T05:32:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:10.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:10 smithi136 ceph-mon[29560]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T05:32:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:10.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:10.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:10 smithi164 ceph-mon[33554]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T05:32:10.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:10.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[29560]: pgmap v690: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[29560]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T05:32:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[38106]: pgmap v690: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:11.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[38106]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T05:32:11.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:11.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:11.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:11 smithi164 ceph-mon[33554]: pgmap v690: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:12.008 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:12.008 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:11 smithi164 ceph-mon[33554]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T05:32:12.009 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:12.009 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:12.009 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:12.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:32:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:32:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:12 smithi164 ceph-mon[33554]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T05:32:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:12.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[38106]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T05:32:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:12.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:12.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[29560]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T05:32:12.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:12.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:12.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:12.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:12.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[38106]: pgmap v693: 260 pgs: 260 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-03-26T05:32:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[38106]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[38106]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[29560]: pgmap v693: 260 pgs: 260 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[29560]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T05:32:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:13.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:13.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[29560]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T05:32:13.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:13.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:13 smithi164 ceph-mon[33554]: pgmap v693: 260 pgs: 260 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-03-26T05:32:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:13 smithi164 ceph-mon[33554]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T05:32:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:13 smithi164 ceph-mon[33554]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T05:32:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:15 smithi136 ceph-mon[38106]: pgmap v696: 260 pgs: 260 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-03-26T05:32:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:15 smithi136 ceph-mon[38106]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T05:32:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:15 smithi136 ceph-mon[29560]: pgmap v696: 260 pgs: 260 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-03-26T05:32:15.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:15.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:15 smithi136 ceph-mon[29560]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T05:32:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:15 smithi164 ceph-mon[33554]: pgmap v696: 260 pgs: 260 active+clean; 8.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-03-26T05:32:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:15 smithi164 ceph-mon[33554]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T05:32:16.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:16 smithi136 ceph-mon[38106]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T05:32:16.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:16 smithi136 ceph-mon[29560]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T05:32:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:16 smithi164 ceph-mon[33554]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T05:32:17.570 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:32:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[38106]: pgmap v699: 260 pgs: 260 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[38106]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-83"}]: dispatch 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-83"}]: dispatch 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[29560]: pgmap v699: 260 pgs: 260 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[29560]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:17.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:17.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-83"}]: dispatch 2024-03-26T05:32:17.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-83"}]: dispatch 2024-03-26T05:32:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:17 smithi164 ceph-mon[33554]: pgmap v699: 260 pgs: 260 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-26T05:32:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:17 smithi164 ceph-mon[33554]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T05:32:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-83"}]: dispatch 2024-03-26T05:32:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-83"}]: dispatch 2024-03-26T05:32:18.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:18 smithi164 ceph-mon[33554]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T05:32:19.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:18 smithi136 ceph-mon[38106]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T05:32:19.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:18 smithi136 ceph-mon[29560]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T05:32:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:19 smithi164 ceph-mon[33554]: pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:19 smithi164 ceph-mon[33554]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T05:32:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:19 smithi136 ceph-mon[38106]: pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:19 smithi136 ceph-mon[38106]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T05:32:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:20.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:19 smithi136 ceph-mon[29560]: pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:20.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:19 smithi136 ceph-mon[29560]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T05:32:20.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:20.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:20.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:20.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:20 smithi164 ceph-mon[33554]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T05:32:20.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:20.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:20.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[38106]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[29560]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:21.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:21 smithi164 ceph-mon[33554]: pgmap v705: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:21 smithi164 ceph-mon[33554]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T05:32:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[38106]: pgmap v705: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[38106]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[29560]: pgmap v705: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[29560]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T05:32:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:22.618 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:32:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-mon[33554]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T05:32:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-mon[33554]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T05:32:22.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[29560]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T05:32:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[29560]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T05:32:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_tier","val": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:32:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[38106]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T05:32:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[38106]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T05:32:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:23 smithi164 ceph-mon[33554]: pgmap v708: 260 pgs: 260 active+clean; 8.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T05:32:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:23 smithi164 ceph-mon[33554]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T05:32:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:23 smithi136 ceph-mon[38106]: pgmap v708: 260 pgs: 260 active+clean; 8.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T05:32:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:23 smithi136 ceph-mon[38106]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T05:32:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:23 smithi136 ceph-mon[29560]: pgmap v708: 260 pgs: 260 active+clean; 8.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T05:32:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:23 smithi136 ceph-mon[29560]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T05:32:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:25 smithi164 ceph-mon[33554]: pgmap v711: 260 pgs: 260 active+clean; 8.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T05:32:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:25 smithi164 ceph-mon[33554]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T05:32:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:25 smithi136 ceph-mon[38106]: pgmap v711: 260 pgs: 260 active+clean; 8.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T05:32:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:25 smithi136 ceph-mon[38106]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T05:32:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:25 smithi136 ceph-mon[29560]: pgmap v711: 260 pgs: 260 active+clean; 8.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T05:32:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:25 smithi136 ceph-mon[29560]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T05:32:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:26 smithi164 ceph-mon[33554]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T05:32:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-85"}]: dispatch 2024-03-26T05:32:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-85"}]: dispatch 2024-03-26T05:32:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[38106]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T05:32:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-85"}]: dispatch 2024-03-26T05:32:27.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-85"}]: dispatch 2024-03-26T05:32:27.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[29560]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T05:32:27.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-85"}]: dispatch 2024-03-26T05:32:27.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-85"}]: dispatch 2024-03-26T05:32:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:32:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:32:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:32:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:27 smithi164 ceph-mon[33554]: pgmap v714: 260 pgs: 260 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 7.5 KiB/s wr, 12 op/s 2024-03-26T05:32:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:27 smithi164 ceph-mon[33554]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T05:32:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:27 smithi136 ceph-mon[29560]: pgmap v714: 260 pgs: 260 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 7.5 KiB/s wr, 12 op/s 2024-03-26T05:32:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:27 smithi136 ceph-mon[29560]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T05:32:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:27 smithi136 ceph-mon[38106]: pgmap v714: 260 pgs: 260 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 7.5 KiB/s wr, 12 op/s 2024-03-26T05:32:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:27 smithi136 ceph-mon[38106]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T05:32:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:28.668 INFO:tasks.workunit.client.0.smithi136.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18096 ms) 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22115 ms) 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3037 ms) 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3020 ms) 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3017 ms) 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.669 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3035 ms) 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4025 ms) 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38530 ms) 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17069 ms) 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4032 ms) 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-26T05:32:28.670 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3024 ms) 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25072 ms) 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15062 ms) 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5033 ms) 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4030 ms) 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.671 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: done waiting 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24575 ms) 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10075 ms) 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9050 ms) 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9073 ms) 2024-03-26T05:32:28.672 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-26T05:32:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:28 smithi164 ceph-mon[33554]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T05:32:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:28 smithi136 ceph-mon[38106]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T05:32:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:28 smithi136 ceph-mon[29560]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T05:32:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:29 smithi164 ceph-mon[33554]: pgmap v717: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T05:32:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:29 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:29 smithi164 ceph-mon[33554]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T05:32:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[29560]: pgmap v717: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T05:32:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[29560]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T05:32:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[38106]: pgmap v717: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T05:32:30.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:30.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:30.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[38106]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T05:32:30.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:30 smithi164 ceph-mon[33554]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T05:32:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:30.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:30 smithi136 ceph-mon[38106]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T05:32:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:31.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:30 smithi136 ceph-mon[29560]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T05:32:31.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:31.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:31 smithi164 ceph-mon[33554]: pgmap v720: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:31 smithi164 ceph-mon[33554]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T05:32:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]: dispatch 2024-03-26T05:32:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]: dispatch 2024-03-26T05:32:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[38106]: pgmap v720: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[38106]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T05:32:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]: dispatch 2024-03-26T05:32:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]: dispatch 2024-03-26T05:32:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[29560]: pgmap v720: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[29560]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T05:32:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]: dispatch 2024-03-26T05:32:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]: dispatch 2024-03-26T05:32:32.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:32:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:32:32.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]': finished 2024-03-26T05:32:32.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:32.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:32 smithi164 ceph-mon[33554]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T05:32:32.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]': finished 2024-03-26T05:32:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:32 smithi136 ceph-mon[38106]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T05:32:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_tier","val": "test-rados-api-smithi136-75071-89-test-flush"}]': finished 2024-03-26T05:32:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:32 smithi136 ceph-mon[29560]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T05:32:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:32:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:33 smithi164 ceph-mon[33554]: pgmap v723: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:32:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:33 smithi164 ceph-mon[33554]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T05:32:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[38106]: pgmap v723: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:32:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[38106]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T05:32:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[29560]: pgmap v723: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:32:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:32:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[29560]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T05:32:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:32:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:34 smithi164 ceph-mon[33554]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T05:32:35.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:35.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:34 smithi136 ceph-mon[38106]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T05:32:35.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:32:35.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:34 smithi136 ceph-mon[29560]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T05:32:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:35 smithi164 ceph-mon[33554]: pgmap v726: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:32:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:35 smithi164 ceph-mon[33554]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T05:32:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-87"}]: dispatch 2024-03-26T05:32:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-87"}]: dispatch 2024-03-26T05:32:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[38106]: pgmap v726: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:32:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[38106]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T05:32:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-87"}]: dispatch 2024-03-26T05:32:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-87"}]: dispatch 2024-03-26T05:32:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[29560]: pgmap v726: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:32:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[29560]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T05:32:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-87"}]: dispatch 2024-03-26T05:32:36.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-87"}]: dispatch 2024-03-26T05:32:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:36 smithi164 ceph-mon[33554]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T05:32:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:36 smithi164 ceph-mon[33554]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:36 smithi164 ceph-mon[33554]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T05:32:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[38106]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T05:32:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[38106]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[38106]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T05:32:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[29560]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T05:32:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[29560]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[29560]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T05:32:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:32:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:32:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:37 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:37 smithi164 ceph-mon[33554]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T05:32:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[38106]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T05:32:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[29560]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T05:32:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:32:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:39 smithi164 ceph-mon[33554]: pgmap v732: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:39 smithi164 ceph-mon[33554]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T05:32:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:39 smithi136 ceph-mon[38106]: pgmap v732: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:39 smithi136 ceph-mon[38106]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T05:32:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:39 smithi136 ceph-mon[29560]: pgmap v732: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:32:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:32:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:39 smithi136 ceph-mon[29560]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T05:32:41.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:40 smithi136 ceph-mon[29560]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T05:32:41.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:40 smithi136 ceph-mon[38106]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T05:32:41.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:40 smithi164 ceph-mon[33554]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T05:32:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[29560]: pgmap v735: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[29560]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T05:32:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-90"}]: dispatch 2024-03-26T05:32:42.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-90"}]: dispatch 2024-03-26T05:32:42.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[38106]: pgmap v735: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:42.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[38106]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T05:32:42.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:42.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:42.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-90"}]: dispatch 2024-03-26T05:32:42.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-90"}]: dispatch 2024-03-26T05:32:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:41 smithi164 ceph-mon[33554]: pgmap v735: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:41 smithi164 ceph-mon[33554]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T05:32:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-90"}]: dispatch 2024-03-26T05:32:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-90"}]: dispatch 2024-03-26T05:32:42.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:32:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:32:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:42 smithi136 ceph-mon[29560]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T05:32:43.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:42 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:43.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:42 smithi136 ceph-mon[38106]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T05:32:43.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:42 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:42 smithi164 ceph-mon[33554]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T05:32:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:42 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[29560]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:32:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[29560]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T05:32:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[38106]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:32:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[38106]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T05:32:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:43 smithi164 ceph-mon[33554]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:32:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:43 smithi164 ceph-mon[33554]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T05:32:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:44 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:44 smithi136 ceph-mon[38106]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T05:32:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:44 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:44 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:44 smithi136 ceph-mon[29560]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T05:32:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:44 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:44 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:44 smithi164 ceph-mon[33554]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T05:32:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:44 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:45 smithi136 ceph-mon[38106]: pgmap v741: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:32:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:45 smithi136 ceph-mon[38106]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T05:32:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:45 smithi136 ceph-mon[29560]: pgmap v741: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:32:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:45 smithi136 ceph-mon[29560]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T05:32:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:45 smithi164 ceph-mon[33554]: pgmap v741: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T05:32:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:45 smithi164 ceph-mon[33554]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T05:32:47.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[38106]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T05:32:47.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:47.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:47.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-92"}]: dispatch 2024-03-26T05:32:47.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-92"}]: dispatch 2024-03-26T05:32:47.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[29560]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T05:32:47.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:47.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:47.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-92"}]: dispatch 2024-03-26T05:32:47.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-92"}]: dispatch 2024-03-26T05:32:47.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:46 smithi164 ceph-mon[33554]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T05:32:47.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:47.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:32:47.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-92"}]: dispatch 2024-03-26T05:32:47.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-92"}]: dispatch 2024-03-26T05:32:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:32:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:32:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:32:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:47 smithi136 ceph-mon[38106]: pgmap v744: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-26T05:32:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:47 smithi136 ceph-mon[38106]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T05:32:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:47 smithi136 ceph-mon[29560]: pgmap v744: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-26T05:32:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:47 smithi136 ceph-mon[29560]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T05:32:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:47 smithi164 ceph-mon[33554]: pgmap v744: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-26T05:32:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:47 smithi164 ceph-mon[33554]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T05:32:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:48 smithi136 ceph-mon[38106]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T05:32:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:49.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:48 smithi136 ceph-mon[29560]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T05:32:49.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:49.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:48 smithi164 ceph-mon[33554]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T05:32:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:32:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[38106]: pgmap v747: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:32:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[38106]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T05:32:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:50.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[29560]: pgmap v747: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:32:50.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:50.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[29560]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T05:32:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:49 smithi164 ceph-mon[33554]: pgmap v747: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:32:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:49 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:32:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:49 smithi164 ceph-mon[33554]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T05:32:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:32:51.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:50 smithi136 ceph-mon[38106]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T05:32:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:50 smithi136 ceph-mon[29560]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T05:32:51.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:50 smithi164 ceph-mon[33554]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T05:32:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:51 smithi136 ceph-mon[38106]: pgmap v750: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:51 smithi136 ceph-mon[38106]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T05:32:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:51 smithi136 ceph-mon[29560]: pgmap v750: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:51 smithi136 ceph-mon[29560]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T05:32:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:51 smithi164 ceph-mon[33554]: pgmap v750: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:32:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:51 smithi164 ceph-mon[33554]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T05:32:52.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:32:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:32:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:53 smithi136 ceph-mon[38106]: pgmap v752: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-26T05:32:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:53 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:32:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:53 smithi136 ceph-mon[29560]: pgmap v752: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-26T05:32:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:53 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:32:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:53 smithi164 ceph-mon[33554]: pgmap v752: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-26T05:32:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:32:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:53 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:32:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:32:55.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:54 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:32:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:32:55.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:32:55.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:54 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:32:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:55 smithi136 ceph-mon[38106]: pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T05:32:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:55 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi136 to 3248M 2024-03-26T05:32:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:55 smithi136 ceph-mon[38106]: Adjusting osd_memory_target on smithi164 to 3248M 2024-03-26T05:32:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:55 smithi136 ceph-mon[29560]: pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T05:32:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:55 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi136 to 3248M 2024-03-26T05:32:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:55 smithi136 ceph-mon[29560]: Adjusting osd_memory_target on smithi164 to 3248M 2024-03-26T05:32:56.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:55 smithi164 ceph-mon[33554]: pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T05:32:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:55 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi136 to 3248M 2024-03-26T05:32:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:55 smithi164 ceph-mon[33554]: Adjusting osd_memory_target on smithi164 to 3248M 2024-03-26T05:32:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:32:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:32:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:32:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:57 smithi136 ceph-mon[38106]: pgmap v754: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T05:32:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:57 smithi136 ceph-mon[29560]: pgmap v754: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T05:32:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:32:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:57 smithi164 ceph-mon[33554]: pgmap v754: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T05:32:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:32:59 smithi136 ceph-mon[38106]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 996 B/s wr, 4 op/s 2024-03-26T05:33:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:32:59 smithi136 ceph-mon[29560]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 996 B/s wr, 4 op/s 2024-03-26T05:33:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:32:59 smithi164 ceph-mon[33554]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 996 B/s wr, 4 op/s 2024-03-26T05:33:02.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:01 smithi164 ceph-mon[33554]: pgmap v756: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T05:33:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:01 smithi136 ceph-mon[38106]: pgmap v756: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T05:33:02.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:01 smithi136 ceph-mon[29560]: pgmap v756: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T05:33:02.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:33:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:33:03.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:02 smithi164 ceph-mon[33554]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T05:33:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:02 smithi136 ceph-mon[38106]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T05:33:03.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:02 smithi136 ceph-mon[29560]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T05:33:03.719 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:33:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=cleanup t=2024-03-26T05:33:03.352304959Z level=info msg="Completed cleanup jobs" duration=2.478012ms 2024-03-26T05:33:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:03 smithi164 ceph-mon[33554]: pgmap v758: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:33:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:03 smithi136 ceph-mon[38106]: pgmap v758: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:33:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:03 smithi136 ceph-mon[29560]: pgmap v758: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:33:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:05 smithi164 ceph-mon[33554]: pgmap v759: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:33:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:05 smithi136 ceph-mon[29560]: pgmap v759: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:33:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:05 smithi136 ceph-mon[38106]: pgmap v759: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:33:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:33:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:33:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:33:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:07 smithi164 ceph-mon[33554]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:08.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:07 smithi136 ceph-mon[29560]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:08.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:07 smithi136 ceph-mon[38106]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:10.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:09 smithi136 ceph-mon[29560]: pgmap v761: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:10.128 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:09 smithi136 ceph-mon[29560]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T05:33:10.128 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:09 smithi136 ceph-mon[38106]: pgmap v761: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:10.128 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:09 smithi136 ceph-mon[38106]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T05:33:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:09 smithi164 ceph-mon[33554]: pgmap v761: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:09 smithi164 ceph-mon[33554]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T05:33:11.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:10 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]: dispatch 2024-03-26T05:33:11.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:10 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]: dispatch 2024-03-26T05:33:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:10 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]: dispatch 2024-03-26T05:33:11.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:10 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]: dispatch 2024-03-26T05:33:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:10 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]: dispatch 2024-03-26T05:33:11.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:10 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]: dispatch 2024-03-26T05:33:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:11 smithi164 ceph-mon[33554]: pgmap v763: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]': finished 2024-03-26T05:33:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:11 smithi164 ceph-mon[33554]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T05:33:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:11 smithi136 ceph-mon[38106]: pgmap v763: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]': finished 2024-03-26T05:33:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:11 smithi136 ceph-mon[38106]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T05:33:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:11 smithi136 ceph-mon[29560]: pgmap v763: 260 pgs: 260 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.17", "id": [5, 3]}]': finished 2024-03-26T05:33:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:11 smithi136 ceph-mon[29560]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T05:33:12.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:33:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:33:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:12 smithi164 ceph-mon[33554]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T05:33:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-94"}]: dispatch 2024-03-26T05:33:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-94"}]: dispatch 2024-03-26T05:33:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[29560]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T05:33:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-94"}]: dispatch 2024-03-26T05:33:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-94"}]: dispatch 2024-03-26T05:33:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[38106]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T05:33:13.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:13.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:13.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-94"}]: dispatch 2024-03-26T05:33:13.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-94"}]: dispatch 2024-03-26T05:33:13.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:13 smithi164 ceph-mon[33554]: pgmap v766: 260 pgs: 1 peering, 259 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:33:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:13 smithi164 ceph-mon[33554]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T05:33:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:13 smithi136 ceph-mon[38106]: pgmap v766: 260 pgs: 1 peering, 259 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:33:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:13 smithi136 ceph-mon[38106]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T05:33:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:13 smithi136 ceph-mon[29560]: pgmap v766: 260 pgs: 1 peering, 259 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:33:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:13 smithi136 ceph-mon[29560]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T05:33:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:14 smithi164 ceph-mon[33554]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T05:33:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:14 smithi164 ceph-mon[33554]: pgmap v769: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:14 smithi136 ceph-mon[38106]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T05:33:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:14 smithi136 ceph-mon[38106]: pgmap v769: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:14 smithi136 ceph-mon[29560]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T05:33:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:14 smithi136 ceph-mon[29560]: pgmap v769: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:15 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:15 smithi164 ceph-mon[33554]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T05:33:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:33:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:15 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:15 smithi136 ceph-mon[38106]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T05:33:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:33:16.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:15 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:16.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:16.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:15 smithi136 ceph-mon[29560]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T05:33:16.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:33:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:16 smithi164 ceph-mon[33554]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T05:33:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:16 smithi164 ceph-mon[33554]: pgmap v772: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:17.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:16 smithi136 ceph-mon[38106]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T05:33:17.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:16 smithi136 ceph-mon[38106]: pgmap v772: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:17.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:16 smithi136 ceph-mon[29560]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T05:33:17.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:16 smithi136 ceph-mon[29560]: pgmap v772: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:33:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:33:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:33:19.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:19 smithi164 ceph-mon[33554]: pgmap v773: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 994 B/s wr, 2 op/s 2024-03-26T05:33:19.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:19 smithi136 ceph-mon[38106]: pgmap v773: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 994 B/s wr, 2 op/s 2024-03-26T05:33:19.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:19 smithi136 ceph-mon[29560]: pgmap v773: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 994 B/s wr, 2 op/s 2024-03-26T05:33:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:21 smithi164 ceph-mon[33554]: pgmap v774: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 833 B/s wr, 2 op/s 2024-03-26T05:33:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:21 smithi136 ceph-mon[38106]: pgmap v774: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 833 B/s wr, 2 op/s 2024-03-26T05:33:21.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:21 smithi136 ceph-mon[29560]: pgmap v774: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 833 B/s wr, 2 op/s 2024-03-26T05:33:22.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:33:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:33:23.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:23 smithi164 ceph-mon[33554]: pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:33:23.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:23 smithi136 ceph-mon[38106]: pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:33:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:23 smithi136 ceph-mon[29560]: pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:33:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:24.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:24 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:24 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:24.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:24 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:25.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:25 smithi164 ceph-mon[33554]: pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 672 B/s wr, 2 op/s 2024-03-26T05:33:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:25 smithi136 ceph-mon[38106]: pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 672 B/s wr, 2 op/s 2024-03-26T05:33:25.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:25 smithi136 ceph-mon[29560]: pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 672 B/s wr, 2 op/s 2024-03-26T05:33:26.219 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:33:25 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:33:25.926+0000 7f2999308700 -1 snap_mapper.add_oid found existing snaps mapped on 101:1d84c4b1:test-rados-api-smithi136-75071-97::foo:21, removing 2024-03-26T05:33:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-96"}]: dispatch 2024-03-26T05:33:26.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-96"}]: dispatch 2024-03-26T05:33:26.220 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:33:25 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:33:25.926+0000 7fd546ac1700 -1 snap_mapper.add_oid found existing snaps mapped on 101:1d84c4b1:test-rados-api-smithi136-75071-97::foo:21, removing 2024-03-26T05:33:26.325 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:33:25 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3[59564]: 2024-03-26T05:33:25.926+0000 7f64b7d30700 -1 snap_mapper.add_oid found existing snaps mapped on 101:1d84c4b1:test-rados-api-smithi136-75071-97::foo:21, removing 2024-03-26T05:33:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-96"}]: dispatch 2024-03-26T05:33:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-96"}]: dispatch 2024-03-26T05:33:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-96"}]: dispatch 2024-03-26T05:33:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-96"}]: dispatch 2024-03-26T05:33:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:27 smithi164 ceph-mon[33554]: pgmap v777: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 606 B/s wr, 2 op/s 2024-03-26T05:33:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:27 smithi164 ceph-mon[33554]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T05:33:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:33:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:33:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:33:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:27 smithi136 ceph-mon[38106]: pgmap v777: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 606 B/s wr, 2 op/s 2024-03-26T05:33:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:27 smithi136 ceph-mon[38106]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T05:33:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:27 smithi136 ceph-mon[29560]: pgmap v777: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 606 B/s wr, 2 op/s 2024-03-26T05:33:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:27 smithi136 ceph-mon[29560]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T05:33:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:28 smithi164 ceph-mon[33554]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T05:33:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:28 smithi164 ceph-mon[33554]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[38106]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[38106]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[29560]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:28.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:28.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:28.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:28 smithi136 ceph-mon[29560]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T05:33:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:29 smithi164 ceph-mon[33554]: pgmap v780: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:33:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:29 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:33:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:29 smithi164 ceph-mon[33554]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T05:33:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[38106]: pgmap v780: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:33:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:33:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[38106]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T05:33:29.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[29560]: pgmap v780: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:33:29.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:29.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:29.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:29.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:33:29.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:29 smithi136 ceph-mon[29560]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T05:33:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]: dispatch 2024-03-26T05:33:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]: dispatch 2024-03-26T05:33:30.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]: dispatch 2024-03-26T05:33:30.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]: dispatch 2024-03-26T05:33:30.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]: dispatch 2024-03-26T05:33:30.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]: dispatch 2024-03-26T05:33:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:31 smithi164 ceph-mon[33554]: pgmap v783: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:31 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:33:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]': finished 2024-03-26T05:33:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:31 smithi164 ceph-mon[33554]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T05:33:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[38106]: pgmap v783: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:33:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]': finished 2024-03-26T05:33:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[38106]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T05:33:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:31.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[29560]: pgmap v783: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:33:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-98", "mode": "writeback"}]': finished 2024-03-26T05:33:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[29560]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T05:33:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:32.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]': finished 2024-03-26T05:33:32.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:33:32.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:32 smithi164 ceph-mon[33554]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T05:33:32.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:33:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]': finished 2024-03-26T05:33:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:33:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:32 smithi136 ceph-mon[38106]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T05:33:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:33:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-98"}]': finished 2024-03-26T05:33:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:33:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:32 smithi136 ceph-mon[29560]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T05:33:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:33:32.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:33:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:33:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:33 smithi164 ceph-mon[33554]: pgmap v786: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:33:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:33 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:33:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:33:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:33 smithi164 ceph-mon[33554]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T05:33:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T05:33:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T05:33:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[38106]: pgmap v786: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:33:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:33:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:33:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[38106]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T05:33:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[29560]: pgmap v786: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[29560]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T05:33:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:34.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T05:33:34.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:34 smithi164 ceph-mon[33554]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T05:33:34.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:34.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:34.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:34 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T05:33:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[38106]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T05:33:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:34.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T05:33:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[29560]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T05:33:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:34 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:35 smithi164 ceph-mon[33554]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:33:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:33:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T05:33:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:35 smithi164 ceph-mon[33554]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T05:33:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T05:33:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[38106]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:33:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:33:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T05:33:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[38106]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T05:33:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T05:33:35.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[29560]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:33:35.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:33:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T05:33:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[29560]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T05:33:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T05:33:36.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T05:33:36.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:36 smithi164 ceph-mon[33554]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T05:33:36.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:36.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:36.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T05:33:36.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:36 smithi136 ceph-mon[38106]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T05:33:36.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:36.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:36.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T05:33:36.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:36 smithi136 ceph-mon[29560]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T05:33:36.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:36.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:37 smithi164 ceph-mon[33554]: pgmap v792: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:33:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:37 smithi164 ceph-mon[33554]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T05:33:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:33:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:33:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[38106]: pgmap v792: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:33:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[38106]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T05:33:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:37.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[29560]: pgmap v792: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:37.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:33:37.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:37.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[29560]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T05:33:37.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]: dispatch 2024-03-26T05:33:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]': finished 2024-03-26T05:33:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:38 smithi164 ceph-mon[33554]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T05:33:38.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]': finished 2024-03-26T05:33:38.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:38 smithi136 ceph-mon[38106]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T05:33:38.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-98"}]': finished 2024-03-26T05:33:38.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:38 smithi136 ceph-mon[29560]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T05:33:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:39 smithi164 ceph-mon[33554]: pgmap v795: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:39 smithi164 ceph-mon[33554]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T05:33:39.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:39 smithi136 ceph-mon[38106]: pgmap v795: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:39.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:39 smithi136 ceph-mon[38106]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T05:33:39.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:39 smithi136 ceph-mon[29560]: pgmap v795: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:39.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:39 smithi136 ceph-mon[29560]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T05:33:40.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:40 smithi164 ceph-mon[33554]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T05:33:40.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:40.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:40.476 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:40 smithi136 ceph-mon[38106]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T05:33:40.476 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:40.476 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:40.477 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:40 smithi136 ceph-mon[29560]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T05:33:40.477 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:40.477 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[29560]: pgmap v798: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[29560]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T05:33:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[38106]: pgmap v798: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[38106]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T05:33:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:41 smithi164 ceph-mon[33554]: pgmap v798: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:41 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:41 smithi164 ceph-mon[33554]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T05:33:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:41.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:33:42.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:33:42.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[38106]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T05:33:42.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:42.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:42.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]': finished 2024-03-26T05:33:42.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[38106]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T05:33:42.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:33:42.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[29560]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T05:33:42.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:42.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:42.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]': finished 2024-03-26T05:33:42.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:42 smithi136 ceph-mon[29560]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T05:33:42.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:33:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:33:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:33:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:42 smithi164 ceph-mon[33554]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T05:33:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:42.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:42.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-6", "overlaypool": "test-rados-api-smithi136-75071-100"}]': finished 2024-03-26T05:33:42.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:42 smithi164 ceph-mon[33554]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T05:33:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: pgmap v801: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]: dispatch 2024-03-26T05:33:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]: dispatch 2024-03-26T05:33:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:33:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]': finished 2024-03-26T05:33:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: pgmap v801: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]: dispatch 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]: dispatch 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]': finished 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:33:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:33:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: pgmap v801: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]: dispatch 2024-03-26T05:33:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]: dispatch 2024-03-26T05:33:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:33:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-100", "mode": "writeback"}]': finished 2024-03-26T05:33:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T05:33:43.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:33:43.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:33:45.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[38106]: pgmap v804: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:45.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:33:45.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[38106]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T05:33:45.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:45.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:45.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[29560]: pgmap v804: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:45.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:33:45.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[29560]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T05:33:45.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:45.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:45 smithi164 ceph-mon[33554]: pgmap v804: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:33:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:45 smithi164 ceph-mon[33554]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T05:33:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:33:46.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:33:46.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:46 smithi136 ceph-mon[38106]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T05:33:46.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:33:46.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:33:46.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:33:46.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:46 smithi136 ceph-mon[29560]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T05:33:46.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:33:46.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:33:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:33:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:46 smithi164 ceph-mon[33554]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T05:33:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:33:46.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:33:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:33:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:33:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[38106]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[38106]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[29560]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[29560]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T05:33:47.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T05:33:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:47 smithi164 ceph-mon[33554]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:47 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:33:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:33:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T05:33:47.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:47 smithi164 ceph-mon[33554]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T05:33:47.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T05:33:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T05:33:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:48 smithi136 ceph-mon[38106]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T05:33:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:48.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:48.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T05:33:48.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:48 smithi136 ceph-mon[29560]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T05:33:48.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:48.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T05:33:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:48 smithi164 ceph-mon[33554]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T05:33:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:48.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:49.255 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[38106]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:49.255 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:33:49.255 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:49.255 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[38106]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T05:33:49.255 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:49.256 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[29560]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:49.256 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:33:49.256 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:49.256 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[29560]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T05:33:49.256 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:49.256 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:49 smithi164 ceph-mon[33554]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:33:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]': finished 2024-03-26T05:33:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:49 smithi164 ceph-mon[33554]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T05:33:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]: dispatch 2024-03-26T05:33:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:49 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:50.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]': finished 2024-03-26T05:33:50.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:50 smithi136 ceph-mon[29560]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T05:33:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:49 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]': finished 2024-03-26T05:33:50.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:50 smithi136 ceph-mon[38106]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T05:33:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-100"}]': finished 2024-03-26T05:33:50.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:50 smithi164 ceph-mon[33554]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T05:33:51.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:51 smithi136 ceph-mon[38106]: pgmap v813: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:51.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:51 smithi136 ceph-mon[38106]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T05:33:51.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:51 smithi136 ceph-mon[29560]: pgmap v813: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:51.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:51 smithi136 ceph-mon[29560]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T05:33:51.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:51 smithi164 ceph-mon[33554]: pgmap v813: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:51.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:51 smithi164 ceph-mon[33554]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T05:33:52.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:52 smithi136 ceph-mon[38106]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T05:33:52.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:52.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:52.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:52 smithi136 ceph-mon[29560]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T05:33:52.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:52.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:52.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:52 smithi164 ceph-mon[33554]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T05:33:52.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:52.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:33:52.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:33:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:33:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[38106]: pgmap v816: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[38106]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T05:33:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:33:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:33:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:33:53.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[29560]: pgmap v816: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[29560]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T05:33:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:33:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:33:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:33:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:53 smithi164 ceph-mon[33554]: pgmap v816: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:33:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:53 smithi164 ceph-mon[33554]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T05:33:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T05:33:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:33:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T05:33:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:33:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:33:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[38106]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T05:33:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:33:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:33:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:33:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:33:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[29560]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T05:33:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:33:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:33:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:54 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:33:54.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T05:33:54.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:54 smithi164 ceph-mon[33554]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T05:33:54.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:33:54.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T05:33:54.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:54 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:54.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:54 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:33:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[38106]: pgmap v819: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:33:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[38106]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T05:33:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:33:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:33:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:33:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[29560]: pgmap v819: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[29560]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:33:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:55 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:33:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:55 smithi164 ceph-mon[33554]: pgmap v819: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:33:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T05:33:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:55 smithi164 ceph-mon[33554]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T05:33:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:33:55.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T05:33:55.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:55 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:33:55.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:55 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:33:55.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:55 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:33:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:33:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[38106]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T05:33:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-102"}]: dispatch 2024-03-26T05:33:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-102"}]: dispatch 2024-03-26T05:33:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:33:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[29560]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T05:33:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-102"}]: dispatch 2024-03-26T05:33:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-102"}]: dispatch 2024-03-26T05:33:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T05:33:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:56 smithi164 ceph-mon[33554]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T05:33:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-6"}]: dispatch 2024-03-26T05:33:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/1832463676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-102"}]: dispatch 2024-03-26T05:33:56.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-6", "tierpool": "test-rados-api-smithi136-75071-102"}]: dispatch 2024-03-26T05:33:57.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:57 smithi136 ceph-mon[38106]: pgmap v822: 260 pgs: 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T05:33:57.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:57 smithi136 ceph-mon[38106]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T05:33:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:57 smithi136 ceph-mon[29560]: pgmap v822: 260 pgs: 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T05:33:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:57 smithi136 ceph-mon[29560]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T05:33:57.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:33:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:33:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:33:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:57 smithi164 ceph-mon[33554]: pgmap v822: 260 pgs: 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T05:33:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:57 smithi164 ceph-mon[33554]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T05:33:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[38106]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T05:33:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[29560]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:33:58.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:58 smithi164 ceph-mon[33554]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T05:33:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:33:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:33:58.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:33:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[38106]: pgmap v825: 196 pgs: 196 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:33:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[38106]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T05:33:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[29560]: pgmap v825: 196 pgs: 196 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:59.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:33:59.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[29560]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T05:33:59.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:59.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:59.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:33:59 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:33:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:59 smithi164 ceph-mon[33554]: pgmap v825: 196 pgs: 196 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:33:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:33:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:59 smithi164 ceph-mon[33554]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T05:33:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:59 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:33:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:33:59 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:00 smithi136 ceph-mon[29560]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T05:34:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:00 smithi136 ceph-mon[29560]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T05:34:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:00 smithi136 ceph-mon[38106]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T05:34:00.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:00.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:00 smithi136 ceph-mon[38106]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T05:34:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:00 smithi164 ceph-mon[33554]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T05:34:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi136-75071-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:00 smithi164 ceph-mon[33554]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T05:34:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:01 smithi164 ceph-mon[33554]: pgmap v828: 196 pgs: 196 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:01 smithi164 ceph-mon[33554]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T05:34:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:01 smithi136 ceph-mon[29560]: pgmap v828: 196 pgs: 196 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:01 smithi136 ceph-mon[29560]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T05:34:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:01 smithi136 ceph-mon[38106]: pgmap v828: 196 pgs: 196 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:01 smithi136 ceph-mon[38106]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T05:34:02.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:34:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:34:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:03 smithi164 ceph-mon[33554]: pgmap v831: 204 pgs: 8 creating+peering, 196 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:03 smithi164 ceph-mon[33554]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T05:34:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:03 smithi136 ceph-mon[29560]: pgmap v831: 204 pgs: 8 creating+peering, 196 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:03 smithi136 ceph-mon[29560]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T05:34:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:03 smithi136 ceph-mon[38106]: pgmap v831: 204 pgs: 8 creating+peering, 196 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:03 smithi136 ceph-mon[38106]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T05:34:04.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:04.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:04 smithi164 ceph-mon[33554]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T05:34:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:04 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:04 smithi164 ceph-mon[33554]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T05:34:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:05.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:05.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[29560]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T05:34:05.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:05.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[29560]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T05:34:05.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:05.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:05.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[38106]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T05:34:05.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:05.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[38106]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T05:34:05.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:05 smithi164 ceph-mon[33554]: pgmap v834: 236 pgs: 32 unknown, 8 creating+peering, 196 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:05 smithi164 ceph-mon[33554]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T05:34:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:05 smithi136 ceph-mon[29560]: pgmap v834: 236 pgs: 32 unknown, 8 creating+peering, 196 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:05 smithi136 ceph-mon[29560]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T05:34:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:05 smithi136 ceph-mon[38106]: pgmap v834: 236 pgs: 32 unknown, 8 creating+peering, 196 active+clean; 582 KiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:06.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:06.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:05 smithi136 ceph-mon[38106]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T05:34:06.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:34:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:34:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-mon[29560]: pgmap v837: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:34:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-mon[29560]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T05:34:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-107", "overlaypool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:07.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-mon[38106]: pgmap v837: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:34:07.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:07.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-mon[38106]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T05:34:07.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:07 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-107", "overlaypool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:07 smithi164 ceph-mon[33554]: pgmap v837: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:34:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:07 smithi164 ceph-mon[33554]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T05:34:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:07 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-107", "overlaypool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-107", "overlaypool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:08 smithi164 ceph-mon[33554]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T05:34:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:34:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:08 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:08 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-107-cache", "mode": "writeback"}]': finished 2024-03-26T05:34:08.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:08 smithi164 ceph-mon[33554]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T05:34:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-107", "overlaypool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[38106]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T05:34:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:34:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-107-cache", "mode": "writeback"}]': finished 2024-03-26T05:34:08.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[38106]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T05:34:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-107", "overlaypool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[29560]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T05:34:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:34:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-107-cache", "mode": "writeback"}]': finished 2024-03-26T05:34:08.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:08 smithi136 ceph-mon[29560]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T05:34:09.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:09 smithi164 ceph-mon[33554]: pgmap v840: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:34:09.797 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-107"}]: dispatch 2024-03-26T05:34:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:09 smithi136 ceph-mon[38106]: pgmap v840: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:34:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-107"}]: dispatch 2024-03-26T05:34:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:09 smithi136 ceph-mon[29560]: pgmap v840: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:34:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-107"}]: dispatch 2024-03-26T05:34:10.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-107"}]': finished 2024-03-26T05:34:10.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:10 smithi164 ceph-mon[33554]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T05:34:10.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:10.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-107"}]': finished 2024-03-26T05:34:10.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:10 smithi136 ceph-mon[38106]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T05:34:10.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:10.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-107"}]': finished 2024-03-26T05:34:10.727 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:10 smithi136 ceph-mon[29560]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T05:34:10.727 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]: dispatch 2024-03-26T05:34:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:11 smithi164 ceph-mon[33554]: pgmap v843: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:11 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:11 smithi164 ceph-mon[33554]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T05:34:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:11 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-03-26T05:34:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-03-26T05:34:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[38106]: pgmap v843: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[38106]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T05:34:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-03-26T05:34:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-03-26T05:34:11.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[29560]: pgmap v843: 268 pgs: 268 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:11.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:11.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3522976932' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-107", "tierpool": "test-rados-api-smithi136-75071-107-cache"}]': finished 2024-03-26T05:34:11.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[29560]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T05:34:11.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-03-26T05:34:11.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-03-26T05:34:12.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:34:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:34:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:12 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]': finished 2024-03-26T05:34:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:12 smithi164 ceph-mon[33554]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T05:34:12.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:12 smithi164 ceph-mon[33554]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T05:34:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:12 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]': finished 2024-03-26T05:34:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:12 smithi136 ceph-mon[38106]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T05:34:12.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:12 smithi136 ceph-mon[38106]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T05:34:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:12 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]': finished 2024-03-26T05:34:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:12 smithi136 ceph-mon[29560]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T05:34:12.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:12 smithi136 ceph-mon[29560]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T05:34:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:13 smithi164 ceph-mon[33554]: pgmap v846: 236 pgs: 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:13.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:13.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[38106]: pgmap v846: 236 pgs: 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:13.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:13.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:13.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:13.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:13.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:13.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[29560]: pgmap v846: 236 pgs: 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:13.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:13.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:13.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:13.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:13.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:34:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:14 smithi164 ceph-mon[33554]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T05:34:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:34:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:14 smithi136 ceph-mon[38106]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T05:34:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:15.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:34:15.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:14 smithi136 ceph-mon[29560]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T05:34:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:15 smithi164 ceph-mon[33554]: pgmap v849: 204 pgs: 204 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:15 smithi164 ceph-mon[33554]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T05:34:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:15 smithi164 ceph-mon[33554]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T05:34:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:15 smithi136 ceph-mon[38106]: pgmap v849: 204 pgs: 204 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:15 smithi136 ceph-mon[38106]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T05:34:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:15 smithi136 ceph-mon[38106]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T05:34:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:15 smithi136 ceph-mon[29560]: pgmap v849: 204 pgs: 204 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:15 smithi136 ceph-mon[29560]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T05:34:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:15 smithi136 ceph-mon[29560]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T05:34:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:34:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:34:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-mon[38106]: pgmap v852: 212 pgs: 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-mon[38106]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T05:34:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-mon[29560]: pgmap v852: 212 pgs: 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-mon[29560]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T05:34:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:17 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:17 smithi164 ceph-mon[33554]: pgmap v852: 212 pgs: 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:17 smithi164 ceph-mon[33554]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T05:34:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:17 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:18 smithi164 ceph-mon[33554]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T05:34:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:18 smithi164 ceph-mon[33554]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T05:34:18.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-109", "overlaypool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[38106]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T05:34:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:18.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[38106]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T05:34:18.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-109", "overlaypool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:18.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:18.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[29560]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T05:34:18.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:18.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:18.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[29560]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T05:34:18.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-109", "overlaypool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:19 smithi164 ceph-mon[33554]: pgmap v855: 244 pgs: 32 unknown, 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-109", "overlaypool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:19 smithi164 ceph-mon[33554]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T05:34:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:34:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:19 smithi136 ceph-mon[38106]: pgmap v855: 244 pgs: 32 unknown, 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-109", "overlaypool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:19 smithi136 ceph-mon[38106]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T05:34:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:34:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:19 smithi136 ceph-mon[29560]: pgmap v855: 244 pgs: 32 unknown, 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-109", "overlaypool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:19 smithi136 ceph-mon[29560]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T05:34:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T05:34:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:20 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-109-cache", "mode": "writeback"}]': finished 2024-03-26T05:34:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:20 smithi164 ceph-mon[33554]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T05:34:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:34:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:20 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-109-cache", "mode": "writeback"}]': finished 2024-03-26T05:34:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:20 smithi136 ceph-mon[38106]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T05:34:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:34:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:20 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-109-cache", "mode": "writeback"}]': finished 2024-03-26T05:34:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:20 smithi136 ceph-mon[29560]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T05:34:20.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:34:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:21 smithi164 ceph-mon[33554]: pgmap v858: 244 pgs: 32 unknown, 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:34:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:21 smithi164 ceph-mon[33554]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T05:34:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:34:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:21 smithi136 ceph-mon[38106]: pgmap v858: 244 pgs: 32 unknown, 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:34:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:21 smithi136 ceph-mon[38106]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T05:34:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:34:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:21 smithi136 ceph-mon[29560]: pgmap v858: 244 pgs: 32 unknown, 8 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:21.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:34:21.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:21 smithi136 ceph-mon[29560]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T05:34:21.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:34:22.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:34:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:34:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:23 smithi164 ceph-mon[33554]: pgmap v861: 244 pgs: 244 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:34:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:23 smithi164 ceph-mon[33554]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T05:34:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:34:23.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[38106]: pgmap v861: 244 pgs: 244 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:34:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[38106]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T05:34:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:34:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[29560]: pgmap v861: 244 pgs: 244 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:34:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[29560]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T05:34:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:34:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:24 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:34:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:24 smithi164 ceph-mon[33554]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T05:34:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T05:34:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T05:34:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:34:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[38106]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T05:34:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T05:34:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T05:34:24.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:24.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:34:24.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[29560]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T05:34:24.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T05:34:24.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T05:34:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:25 smithi164 ceph-mon[33554]: pgmap v864: 244 pgs: 244 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:25 smithi164 ceph-mon[33554]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T05:34:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:25 smithi136 ceph-mon[38106]: pgmap v864: 244 pgs: 244 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:25 smithi136 ceph-mon[38106]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T05:34:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:25 smithi136 ceph-mon[29560]: pgmap v864: 244 pgs: 244 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:25 smithi136 ceph-mon[29560]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T05:34:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:26 smithi164 ceph-mon[33554]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T05:34:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:26 smithi164 ceph-mon[33554]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T05:34:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[38106]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T05:34:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:26.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[38106]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T05:34:26.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[29560]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T05:34:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]: dispatch 2024-03-26T05:34:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-109", "tierpool": "test-rados-api-smithi136-75071-109-cache"}]': finished 2024-03-26T05:34:26.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:26 smithi136 ceph-mon[29560]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T05:34:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:34:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:34:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:34:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:27 smithi136 ceph-mon[38106]: pgmap v867: 244 pgs: 244 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:34:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:27 smithi136 ceph-mon[38106]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T05:34:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:27 smithi136 ceph-mon[29560]: pgmap v867: 244 pgs: 244 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:34:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:27 smithi136 ceph-mon[29560]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T05:34:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:27 smithi164 ceph-mon[33554]: pgmap v867: 244 pgs: 244 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:34:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:27 smithi164 ceph-mon[33554]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T05:34:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:28 smithi164 ceph-mon[33554]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T05:34:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:28.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:28.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:28 smithi136 ceph-mon[38106]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T05:34:28.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:28 smithi136 ceph-mon[29560]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T05:34:28.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:29 smithi164 ceph-mon[33554]: pgmap v870: 212 pgs: 212 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:29 smithi164 ceph-mon[33554]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T05:34:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:29 smithi136 ceph-mon[38106]: pgmap v870: 212 pgs: 212 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:29 smithi136 ceph-mon[38106]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T05:34:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:29 smithi136 ceph-mon[29560]: pgmap v870: 212 pgs: 212 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:29 smithi136 ceph-mon[29560]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T05:34:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]: dispatch 2024-03-26T05:34:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:31 smithi164 ceph-mon[33554]: pgmap v873: 204 pgs: 204 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:31 smithi164 ceph-mon[33554]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T05:34:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:31 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:31 smithi136 ceph-mon[38106]: pgmap v873: 204 pgs: 204 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:31 smithi136 ceph-mon[38106]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T05:34:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:31 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:31 smithi136 ceph-mon[29560]: pgmap v873: 204 pgs: 204 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/160634002' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-109"}]': finished 2024-03-26T05:34:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:31 smithi136 ceph-mon[29560]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T05:34:31.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:31 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:32.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:34:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:34:32.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:32 smithi164 ceph-mon[33554]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T05:34:32.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:32.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:32.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:32 smithi136 ceph-mon[38106]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T05:34:32.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:32.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:32.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:32 smithi136 ceph-mon[29560]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T05:34:32.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:32.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:33 smithi164 ceph-mon[33554]: pgmap v876: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:34:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:33 smithi164 ceph-mon[33554]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T05:34:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:33 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:33 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:33.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[38106]: pgmap v876: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:34:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[38106]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T05:34:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:33.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[29560]: pgmap v876: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:34:33.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:33.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[29560]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T05:34:33.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3312761545' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:33.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]: dispatch 2024-03-26T05:34:33.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[29560]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T05:34:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:34.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:34.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[38106]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T05:34:34.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:34.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi136-75071-104"}]': finished 2024-03-26T05:34:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:34 smithi164 ceph-mon[33554]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T05:34:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T05:34:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[29560]: pgmap v879: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:34:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:34:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[29560]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T05:34:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:36.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[38106]: pgmap v879: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:34:36.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:34:36.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[38106]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T05:34:36.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:36.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:35 smithi164 ceph-mon[33554]: pgmap v879: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T05:34:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi136-75071-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T05:34:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:35 smithi164 ceph-mon[33554]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T05:34:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:37.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:36 smithi136 ceph-mon[29560]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T05:34:37.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:37.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:36 smithi136 ceph-mon[29560]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T05:34:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:36 smithi136 ceph-mon[38106]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T05:34:37.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:37.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:36 smithi136 ceph-mon[38106]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T05:34:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:36 smithi164 ceph-mon[33554]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T05:34:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi136-75071-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:36 smithi164 ceph-mon[33554]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T05:34:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:34:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:34:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-mon[29560]: pgmap v882: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-mon[29560]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T05:34:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:38.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-mon[38106]: pgmap v882: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:38.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-mon[38106]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T05:34:38.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:38.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:37 smithi164 ceph-mon[33554]: pgmap v882: 196 pgs: 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:37 smithi164 ceph-mon[33554]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T05:34:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[29560]: pgmap v885: 236 pgs: 40 unknown, 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[29560]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T05:34:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:39.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:39.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:39.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[38106]: pgmap v885: 236 pgs: 40 unknown, 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:39.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:39.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[38106]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T05:34:39.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:39.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:39.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:39 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:39 smithi164 ceph-mon[33554]: pgmap v885: 236 pgs: 40 unknown, 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:39 smithi164 ceph-mon[33554]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T05:34:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:39.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:39.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:39 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:40.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:40.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:40 smithi136 ceph-mon[38106]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T05:34:40.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:40.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:40.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:40.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:40 smithi136 ceph-mon[29560]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T05:34:40.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:40.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:40.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:40.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:40 smithi164 ceph-mon[33554]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T05:34:40.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:40.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[38106]: pgmap v888: 236 pgs: 40 unknown, 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]': finished 2024-03-26T05:34:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[38106]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T05:34:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:41.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[29560]: pgmap v888: 236 pgs: 40 unknown, 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]': finished 2024-03-26T05:34:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[29560]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T05:34:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:41.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:41 smithi164 ceph-mon[33554]: pgmap v888: 236 pgs: 40 unknown, 196 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-112"}]': finished 2024-03-26T05:34:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:41 smithi164 ceph-mon[33554]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T05:34:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[38106]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T05:34:42.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:42.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:42.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[29560]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T05:34:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:42.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:42 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:42.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:34:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:34:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:42 smithi164 ceph-mon[33554]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T05:34:42.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:42.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]: dispatch 2024-03-26T05:34:42.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:42 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:43 smithi136 ceph-mon[38106]: pgmap v891: 236 pgs: 236 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:34:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]': finished 2024-03-26T05:34:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:43 smithi136 ceph-mon[38106]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T05:34:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:43 smithi136 ceph-mon[29560]: pgmap v891: 236 pgs: 236 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:34:43.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:43.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]': finished 2024-03-26T05:34:43.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:43 smithi136 ceph-mon[29560]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T05:34:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:43 smithi164 ceph-mon[33554]: pgmap v891: 236 pgs: 236 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T05:34:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:43.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-112"}]': finished 2024-03-26T05:34:43.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:43 smithi164 ceph-mon[33554]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T05:34:44.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:44 smithi136 ceph-mon[38106]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T05:34:44.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:44 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:44.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:44 smithi136 ceph-mon[29560]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T05:34:44.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:44 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:44 smithi164 ceph-mon[33554]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T05:34:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:44 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:34:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:45 smithi136 ceph-mon[38106]: pgmap v894: 204 pgs: 204 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:45 smithi136 ceph-mon[38106]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T05:34:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:45.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:45 smithi136 ceph-mon[29560]: pgmap v894: 204 pgs: 204 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:45.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:45 smithi136 ceph-mon[29560]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T05:34:45.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:45.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:45 smithi164 ceph-mon[33554]: pgmap v894: 204 pgs: 204 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:45 smithi164 ceph-mon[33554]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T05:34:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:46 smithi136 ceph-mon[38106]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T05:34:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:46.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:46.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:46 smithi136 ceph-mon[29560]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T05:34:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:46.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:46.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:46.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:46 smithi164 ceph-mon[33554]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T05:34:46.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:46.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:47.562 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:34:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:34:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[38106]: pgmap v897: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:34:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[38106]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T05:34:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:47.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[29560]: pgmap v897: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:34:47.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:47.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[29560]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T05:34:47.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:47.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:47 smithi164 ceph-mon[33554]: pgmap v897: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:34:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:47 smithi164 ceph-mon[33554]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T05:34:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]': finished 2024-03-26T05:34:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[38106]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T05:34:48.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]: dispatch 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]: dispatch 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]': finished 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[38106]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]': finished 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[29560]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]: dispatch 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]: dispatch 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:48.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]': finished 2024-03-26T05:34:48.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:48 smithi136 ceph-mon[29560]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T05:34:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-114"}]': finished 2024-03-26T05:34:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:48 smithi164 ceph-mon[33554]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T05:34:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]: dispatch 2024-03-26T05:34:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]: dispatch 2024-03-26T05:34:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:48 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:48.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-114", "mode": "writeback"}]': finished 2024-03-26T05:34:48.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:48 smithi164 ceph-mon[33554]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T05:34:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:49 smithi136 ceph-mon[38106]: pgmap v900: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:49.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:49 smithi136 ceph-mon[29560]: pgmap v900: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:49.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:49.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:49 smithi164 ceph-mon[33554]: pgmap v900: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:34:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:34:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:50 smithi164 ceph-mon[33554]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T05:34:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:50.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:51.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:51.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:50 smithi136 ceph-mon[38106]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T05:34:51.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:51.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:34:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:50 smithi136 ceph-mon[29560]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T05:34:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:51.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]: dispatch 2024-03-26T05:34:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:51 smithi164 ceph-mon[33554]: pgmap v903: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:51 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]': finished 2024-03-26T05:34:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:51 smithi164 ceph-mon[33554]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T05:34:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:51 smithi136 ceph-mon[38106]: pgmap v903: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:51 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]': finished 2024-03-26T05:34:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:51 smithi136 ceph-mon[38106]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T05:34:52.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:51 smithi136 ceph-mon[29560]: pgmap v903: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:34:52.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:51 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:34:52.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-114"}]': finished 2024-03-26T05:34:52.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:51 smithi136 ceph-mon[29560]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T05:34:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:52 smithi164 ceph-mon[33554]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T05:34:52.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:34:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:34:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:52 smithi136 ceph-mon[38106]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T05:34:53.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:52 smithi136 ceph-mon[29560]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T05:34:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:53 smithi164 ceph-mon[33554]: pgmap v906: 204 pgs: 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:53 smithi164 ceph-mon[33554]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T05:34:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[38106]: pgmap v906: 204 pgs: 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[38106]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T05:34:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[29560]: pgmap v906: 204 pgs: 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:34:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[29560]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T05:34:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:34:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:54 smithi164 ceph-mon[33554]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T05:34:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:54 smithi136 ceph-mon[38106]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T05:34:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:34:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:54 smithi136 ceph-mon[29560]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T05:34:55.960 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[38106]: pgmap v909: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:55.960 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[38106]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T05:34:55.960 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:55.960 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:55.961 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:34:55.961 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[29560]: pgmap v909: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:55.961 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[29560]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T05:34:55.961 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:55.961 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:55.961 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:55 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:34:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:55 smithi164 ceph-mon[33554]: pgmap v909: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:34:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:55 smithi164 ceph-mon[33554]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T05:34:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:34:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:55 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:34:56.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:56.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:56 smithi164 ceph-mon[33554]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T05:34:56.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:34:56.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:34:56.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]': finished 2024-03-26T05:34:56.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:56 smithi164 ceph-mon[33554]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T05:34:56.990 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:56.990 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[38106]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T05:34:56.990 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:34:56.991 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:34:56.991 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]': finished 2024-03-26T05:34:56.991 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[38106]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T05:34:56.991 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:34:56.991 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[29560]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T05:34:56.992 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:34:56.992 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:34:56.992 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-116"}]': finished 2024-03-26T05:34:56.992 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:56 smithi136 ceph-mon[29560]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T05:34:57.325 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:34:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:34:57.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[38106]: pgmap v912: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:34:57.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]: dispatch 2024-03-26T05:34:57.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]: dispatch 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]': finished 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[29560]: pgmap v912: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]: dispatch 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]: dispatch 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:57.727 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]': finished 2024-03-26T05:34:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:57 smithi164 ceph-mon[33554]: pgmap v912: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:34:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]: dispatch 2024-03-26T05:34:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]: dispatch 2024-03-26T05:34:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:34:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:57 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:34:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-116", "mode": "writeback"}]': finished 2024-03-26T05:34:58.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:58 smithi164 ceph-mon[33554]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T05:34:58.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "313.4"}]: dispatch 2024-03-26T05:34:58.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:58 smithi164 ceph-mon[33554]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "313.4"}]: dispatch 2024-03-26T05:34:58.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:58 smithi164 ceph-mon[33554]: 313.4 scrub starts 2024-03-26T05:34:58.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:58 smithi164 ceph-mon[33554]: 313.4 scrub ok 2024-03-26T05:34:58.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:58 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:34:58.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:58 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:34:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[38106]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T05:34:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "313.4"}]: dispatch 2024-03-26T05:34:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[38106]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "313.4"}]: dispatch 2024-03-26T05:34:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[38106]: 313.4 scrub starts 2024-03-26T05:34:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[38106]: 313.4 scrub ok 2024-03-26T05:34:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:34:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:34:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[29560]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T05:34:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "313.4"}]: dispatch 2024-03-26T05:34:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[29560]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "313.4"}]: dispatch 2024-03-26T05:34:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[29560]: 313.4 scrub starts 2024-03-26T05:34:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[29560]: 313.4 scrub ok 2024-03-26T05:34:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:34:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:58 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:34:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:59 smithi164 ceph-mon[33554]: pgmap v915: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:34:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:59 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:34:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:34:59 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:59 smithi136 ceph-mon[29560]: pgmap v915: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:35:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:59 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:34:59 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:59 smithi136 ceph-mon[38106]: pgmap v915: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:35:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:59 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:00.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:34:59 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:00 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:35:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:00 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:35:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:00 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:00 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:35:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:00 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:35:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:00 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:01.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:00 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:35:01.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:00 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:35:01.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:00 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:02.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:01 smithi164 ceph-mon[33554]: pgmap v916: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T05:35:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:01 smithi136 ceph-mon[38106]: pgmap v916: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T05:35:02.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:01 smithi136 ceph-mon[29560]: pgmap v916: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T05:35:02.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:35:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:35:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:03 smithi164 ceph-mon[33554]: pgmap v917: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-26T05:35:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:03 smithi136 ceph-mon[38106]: pgmap v917: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-26T05:35:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:03 smithi136 ceph-mon[29560]: pgmap v917: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-26T05:35:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:05 smithi164 ceph-mon[33554]: pgmap v918: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-03-26T05:35:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:05 smithi136 ceph-mon[38106]: pgmap v918: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-03-26T05:35:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:05 smithi136 ceph-mon[29560]: pgmap v918: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-03-26T05:35:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:35:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:35:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:35:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:07 smithi164 ceph-mon[33554]: pgmap v919: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:35:08.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:07 smithi136 ceph-mon[38106]: pgmap v919: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:35:08.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:07 smithi136 ceph-mon[29560]: pgmap v919: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:35:10.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:09 smithi136 ceph-mon[38106]: pgmap v920: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:35:10.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:09 smithi136 ceph-mon[29560]: pgmap v920: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:35:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:09 smithi164 ceph-mon[33554]: pgmap v920: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:35:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:11 smithi164 ceph-mon[33554]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T05:35:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:11 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]: dispatch 2024-03-26T05:35:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]: dispatch 2024-03-26T05:35:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:11 smithi136 ceph-mon[38106]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T05:35:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:11 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]: dispatch 2024-03-26T05:35:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]: dispatch 2024-03-26T05:35:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:11 smithi136 ceph-mon[29560]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T05:35:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:11 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]: dispatch 2024-03-26T05:35:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]: dispatch 2024-03-26T05:35:12.875 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:35:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:35:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:12 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]': finished 2024-03-26T05:35:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:12 smithi164 ceph-mon[33554]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T05:35:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:12 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:12 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]': finished 2024-03-26T05:35:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:12 smithi136 ceph-mon[38106]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T05:35:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:12 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:12 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.3", "id": [0, 1]}]': finished 2024-03-26T05:35:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:12 smithi136 ceph-mon[29560]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T05:35:13.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:12 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:13.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:13 smithi164 ceph-mon[33554]: pgmap v923: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:35:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:13 smithi164 ceph-mon[33554]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T05:35:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[38106]: pgmap v923: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:35:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[38106]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T05:35:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[29560]: pgmap v923: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:35:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[29560]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T05:35:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:14 smithi164 ceph-mon[33554]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T05:35:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:35:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:35:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:14 smithi136 ceph-mon[29560]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T05:35:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:35:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:35:15.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:15.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:14 smithi136 ceph-mon[38106]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T05:35:15.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:35:15.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]: dispatch 2024-03-26T05:35:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:15 smithi164 ceph-mon[33554]: pgmap v926: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:35:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:15 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]': finished 2024-03-26T05:35:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:15 smithi164 ceph-mon[33554]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T05:35:16.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:15 smithi136 ceph-mon[38106]: pgmap v926: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:35:16.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:15 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:16.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]': finished 2024-03-26T05:35:16.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:15 smithi136 ceph-mon[38106]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T05:35:16.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:15 smithi136 ceph-mon[29560]: pgmap v926: 236 pgs: 236 active+clean; 582 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:35:16.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:15 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:16.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-116"}]': finished 2024-03-26T05:35:16.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:15 smithi136 ceph-mon[29560]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T05:35:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:16 smithi164 ceph-mon[33554]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T05:35:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:16 smithi164 ceph-mon[33554]: pgmap v929: 204 pgs: 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T05:35:17.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:16 smithi136 ceph-mon[38106]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T05:35:17.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:16 smithi136 ceph-mon[38106]: pgmap v929: 204 pgs: 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T05:35:17.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:16 smithi136 ceph-mon[29560]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T05:35:17.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:16 smithi136 ceph-mon[29560]: pgmap v929: 204 pgs: 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T05:35:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:35:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:35:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:35:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:17 smithi164 ceph-mon[33554]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T05:35:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:17 smithi136 ceph-mon[38106]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T05:35:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:18.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:17 smithi136 ceph-mon[29560]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T05:35:18.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:18.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:18 smithi164 ceph-mon[33554]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T05:35:19.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:18 smithi164 ceph-mon[33554]: pgmap v932: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T05:35:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:18 smithi136 ceph-mon[38106]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T05:35:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:18 smithi136 ceph-mon[38106]: pgmap v932: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T05:35:19.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:19.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:18 smithi136 ceph-mon[29560]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T05:35:19.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:18 smithi136 ceph-mon[29560]: pgmap v932: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T05:35:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:19 smithi164 ceph-mon[33554]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T05:35:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:20.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:20.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:20.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:19 smithi164 ceph-mon[33554]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T05:35:20.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:20.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[38106]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T05:35:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[38106]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T05:35:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[29560]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T05:35:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[29560]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T05:35:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]': finished 2024-03-26T05:35:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:20 smithi164 ceph-mon[33554]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T05:35:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]: dispatch 2024-03-26T05:35:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]: dispatch 2024-03-26T05:35:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:20 smithi164 ceph-mon[33554]: pgmap v936: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]': finished 2024-03-26T05:35:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[38106]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T05:35:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]: dispatch 2024-03-26T05:35:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]: dispatch 2024-03-26T05:35:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[38106]: pgmap v936: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:21.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-118"}]': finished 2024-03-26T05:35:21.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[29560]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T05:35:21.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]: dispatch 2024-03-26T05:35:21.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]: dispatch 2024-03-26T05:35:21.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:20 smithi136 ceph-mon[29560]: pgmap v936: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:21 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]': finished 2024-03-26T05:35:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:21 smithi164 ceph-mon[33554]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T05:35:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:21 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]': finished 2024-03-26T05:35:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:21 smithi136 ceph-mon[38106]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T05:35:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:21 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-118", "mode": "writeback"}]': finished 2024-03-26T05:35:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:21 smithi136 ceph-mon[29560]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T05:35:22.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:35:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:35:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[38106]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T05:35:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[38106]: pgmap v939: 236 pgs: 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 652 B/s wr, 2 op/s 2024-03-26T05:35:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:23.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[29560]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T05:35:23.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[29560]: pgmap v939: 236 pgs: 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 652 B/s wr, 2 op/s 2024-03-26T05:35:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:22 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:23.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:22 smithi164 ceph-mon[33554]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T05:35:23.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:23.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:23.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:22 smithi164 ceph-mon[33554]: pgmap v939: 236 pgs: 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 652 B/s wr, 2 op/s 2024-03-26T05:35:23.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:22 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:23 smithi136 ceph-mon[38106]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T05:35:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:24.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:24.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:23 smithi136 ceph-mon[29560]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T05:35:24.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:24.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:24.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:24.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:23 smithi164 ceph-mon[33554]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T05:35:24.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:24.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]: dispatch 2024-03-26T05:35:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:25 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]': finished 2024-03-26T05:35:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:25 smithi136 ceph-mon[38106]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T05:35:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:25 smithi136 ceph-mon[38106]: pgmap v942: 236 pgs: 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:35:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:25 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]': finished 2024-03-26T05:35:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:25 smithi136 ceph-mon[29560]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T05:35:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:25 smithi136 ceph-mon[29560]: pgmap v942: 236 pgs: 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:35:25.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:25 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:25.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-118"}]': finished 2024-03-26T05:35:25.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:25 smithi164 ceph-mon[33554]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T05:35:25.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:25 smithi164 ceph-mon[33554]: pgmap v942: 236 pgs: 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:35:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:26 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:26 smithi136 ceph-mon[38106]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T05:35:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:26 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:26 smithi136 ceph-mon[29560]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T05:35:26.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:26 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:26.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:26 smithi164 ceph-mon[33554]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T05:35:27.325 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:35:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:35:27.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[38106]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T05:35:27.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:27.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[38106]: pgmap v945: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[38106]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[29560]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[29560]: pgmap v945: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:27.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:27 smithi136 ceph-mon[29560]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T05:35:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:27 smithi164 ceph-mon[33554]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T05:35:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:27 smithi164 ceph-mon[33554]: pgmap v945: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:27 smithi164 ceph-mon[33554]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T05:35:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[38106]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[29560]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:28 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:28.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:28 smithi164 ceph-mon[33554]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T05:35:28.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:28.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[38106]: pgmap v948: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]': finished 2024-03-26T05:35:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[38106]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T05:35:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]: dispatch 2024-03-26T05:35:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]: dispatch 2024-03-26T05:35:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[29560]: pgmap v948: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]': finished 2024-03-26T05:35:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[29560]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T05:35:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]: dispatch 2024-03-26T05:35:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]: dispatch 2024-03-26T05:35:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:29 smithi164 ceph-mon[33554]: pgmap v948: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-120"}]': finished 2024-03-26T05:35:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:29 smithi164 ceph-mon[33554]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T05:35:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:29 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]: dispatch 2024-03-26T05:35:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]: dispatch 2024-03-26T05:35:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]': finished 2024-03-26T05:35:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[38106]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T05:35:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]': finished 2024-03-26T05:35:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[29560]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T05:35:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:30 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-120", "mode": "writeback"}]': finished 2024-03-26T05:35:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:30 smithi164 ceph-mon[33554]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T05:35:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:31.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[38106]: pgmap v951: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:31.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:31.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[38106]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T05:35:31.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:31.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:31.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[29560]: pgmap v951: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:31.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:31.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[29560]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T05:35:31.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:31.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:31 smithi164 ceph-mon[33554]: pgmap v951: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:31 smithi164 ceph-mon[33554]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T05:35:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]: dispatch 2024-03-26T05:35:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:32 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]': finished 2024-03-26T05:35:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:32 smithi136 ceph-mon[38106]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T05:35:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:32 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]': finished 2024-03-26T05:35:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:32 smithi136 ceph-mon[29560]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T05:35:32.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:32 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:32.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-120"}]': finished 2024-03-26T05:35:32.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:32 smithi164 ceph-mon[33554]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T05:35:32.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:35:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:35:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:33 smithi136 ceph-mon[38106]: pgmap v954: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T05:35:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:33 smithi136 ceph-mon[38106]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T05:35:33.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:33 smithi136 ceph-mon[29560]: pgmap v954: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T05:35:33.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:33 smithi136 ceph-mon[29560]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T05:35:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:33 smithi164 ceph-mon[33554]: pgmap v954: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T05:35:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:33 smithi164 ceph-mon[33554]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T05:35:34.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:34 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:34.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:34 smithi136 ceph-mon[29560]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T05:35:34.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:34.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:34 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:34.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:34 smithi136 ceph-mon[38106]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T05:35:34.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:34.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:34 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:34 smithi164 ceph-mon[33554]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T05:35:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[38106]: pgmap v957: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:35:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[38106]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T05:35:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[29560]: pgmap v957: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:35:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[29560]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T05:35:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:35 smithi164 ceph-mon[33554]: pgmap v957: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:35:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:35 smithi164 ceph-mon[33554]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T05:35:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:35.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[38106]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T05:35:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[38106]: pgmap v960: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:35:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[29560]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T05:35:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:37.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:36 smithi136 ceph-mon[29560]: pgmap v960: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:35:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:36 smithi164 ceph-mon[33554]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T05:35:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:36 smithi164 ceph-mon[33554]: pgmap v960: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:35:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:35:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:35:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]': finished 2024-03-26T05:35:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-mon[38106]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T05:35:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]: dispatch 2024-03-26T05:35:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]: dispatch 2024-03-26T05:35:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]': finished 2024-03-26T05:35:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-mon[29560]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T05:35:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]: dispatch 2024-03-26T05:35:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]: dispatch 2024-03-26T05:35:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-122"}]': finished 2024-03-26T05:35:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:37 smithi164 ceph-mon[33554]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T05:35:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]: dispatch 2024-03-26T05:35:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]: dispatch 2024-03-26T05:35:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:39 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]': finished 2024-03-26T05:35:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:39 smithi136 ceph-mon[38106]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T05:35:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:39 smithi136 ceph-mon[38106]: pgmap v963: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:39 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:38 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]': finished 2024-03-26T05:35:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:38 smithi136 ceph-mon[29560]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T05:35:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:38 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:38 smithi136 ceph-mon[29560]: pgmap v963: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:38 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:38 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-122", "mode": "writeback"}]': finished 2024-03-26T05:35:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:38 smithi164 ceph-mon[33554]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T05:35:39.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:38 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:39.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:39.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:38 smithi164 ceph-mon[33554]: pgmap v963: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:39.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:38 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:40 smithi136 ceph-mon[29560]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T05:35:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:40 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:40 smithi136 ceph-mon[38106]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T05:35:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:40 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:40.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:40.371 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:40 smithi164 ceph-mon[33554]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T05:35:40.371 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:40 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:40.371 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]: dispatch 2024-03-26T05:35:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:41 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]': finished 2024-03-26T05:35:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:41 smithi136 ceph-mon[38106]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T05:35:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:41 smithi136 ceph-mon[38106]: pgmap v966: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:41.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:41 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:41.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]': finished 2024-03-26T05:35:41.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:41 smithi136 ceph-mon[29560]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T05:35:41.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:41 smithi136 ceph-mon[29560]: pgmap v966: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:41 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-122"}]': finished 2024-03-26T05:35:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:41 smithi164 ceph-mon[33554]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T05:35:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:41 smithi164 ceph-mon[33554]: pgmap v966: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:42 smithi136 ceph-mon[29560]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T05:35:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:42 smithi136 ceph-mon[38106]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T05:35:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:42 smithi164 ceph-mon[33554]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T05:35:42.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:35:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:35:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[38106]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T05:35:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[38106]: pgmap v969: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[38106]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[29560]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[29560]: pgmap v969: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:43 smithi136 ceph-mon[29560]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T05:35:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:43 smithi164 ceph-mon[33554]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T05:35:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:43 smithi164 ceph-mon[33554]: pgmap v969: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:43 smithi164 ceph-mon[33554]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T05:35:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:44 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:44 smithi136 ceph-mon[38106]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T05:35:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:44 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:44 smithi136 ceph-mon[29560]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T05:35:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:44 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:44 smithi164 ceph-mon[33554]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T05:35:45.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:45.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:45.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:45 smithi136 ceph-mon[38106]: pgmap v972: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:45.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:45 smithi136 ceph-mon[29560]: pgmap v972: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:45 smithi164 ceph-mon[33554]: pgmap v972: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:35:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:46 smithi136 ceph-mon[38106]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T05:35:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:46 smithi136 ceph-mon[29560]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T05:35:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:46 smithi164 ceph-mon[33554]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T05:35:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:47.325 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:35:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:35:47.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]': finished 2024-03-26T05:35:47.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[38106]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T05:35:47.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]: dispatch 2024-03-26T05:35:47.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]: dispatch 2024-03-26T05:35:47.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[38106]: pgmap v975: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:35:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]': finished 2024-03-26T05:35:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[29560]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T05:35:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]: dispatch 2024-03-26T05:35:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]: dispatch 2024-03-26T05:35:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:47 smithi136 ceph-mon[29560]: pgmap v975: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:35:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-124"}]': finished 2024-03-26T05:35:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:47 smithi164 ceph-mon[33554]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T05:35:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]: dispatch 2024-03-26T05:35:47.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]: dispatch 2024-03-26T05:35:47.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:47 smithi164 ceph-mon[33554]: pgmap v975: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:35:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:48 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]': finished 2024-03-26T05:35:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:48 smithi136 ceph-mon[38106]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T05:35:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:48 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]': finished 2024-03-26T05:35:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:48 smithi136 ceph-mon[29560]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T05:35:48.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:48 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:48.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-124", "mode": "writeback"}]': finished 2024-03-26T05:35:48.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:48 smithi164 ceph-mon[33554]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T05:35:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[29560]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T05:35:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[29560]: pgmap v978: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:35:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:49.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:49.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[38106]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T05:35:49.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[38106]: pgmap v978: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:35:49.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:49.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:49.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:49 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:49 smithi164 ceph-mon[33554]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T05:35:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:49 smithi164 ceph-mon[33554]: pgmap v978: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T05:35:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:49 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[29560]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]': finished 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[29560]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[38106]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:50.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]': finished 2024-03-26T05:35:50.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:50 smithi136 ceph-mon[38106]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T05:35:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:50 smithi164 ceph-mon[33554]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T05:35:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]: dispatch 2024-03-26T05:35:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:50 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:50.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-124"}]': finished 2024-03-26T05:35:50.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:50 smithi164 ceph-mon[33554]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T05:35:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:51 smithi164 ceph-mon[33554]: pgmap v981: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:51.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:51 smithi136 ceph-mon[38106]: pgmap v981: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:51.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:51 smithi136 ceph-mon[29560]: pgmap v981: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:35:52.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:52 smithi164 ceph-mon[33554]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T05:35:52.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:52 smithi136 ceph-mon[38106]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T05:35:52.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:52 smithi136 ceph-mon[29560]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T05:35:52.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:35:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:35:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:53 smithi164 ceph-mon[33554]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:53 smithi164 ceph-mon[33554]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T05:35:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[38106]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[38106]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T05:35:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:53.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[29560]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:53.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[29560]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T05:35:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:35:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:35:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:54 smithi164 ceph-mon[33554]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T05:35:54.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:54.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:54.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:54.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:54 smithi164 ceph-mon[33554]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T05:35:54.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:54.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[38106]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T05:35:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:54.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[38106]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[29560]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[29560]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:54.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:55 smithi164 ceph-mon[33554]: pgmap v987: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]': finished 2024-03-26T05:35:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:55 smithi164 ceph-mon[33554]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T05:35:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]: dispatch 2024-03-26T05:35:55.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]: dispatch 2024-03-26T05:35:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[38106]: pgmap v987: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]': finished 2024-03-26T05:35:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[38106]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T05:35:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]: dispatch 2024-03-26T05:35:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]: dispatch 2024-03-26T05:35:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[29560]: pgmap v987: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-126"}]': finished 2024-03-26T05:35:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[29560]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T05:35:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]: dispatch 2024-03-26T05:35:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]: dispatch 2024-03-26T05:35:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:56 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]': finished 2024-03-26T05:35:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:56 smithi164 ceph-mon[33554]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T05:35:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]': finished 2024-03-26T05:35:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[38106]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T05:35:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:35:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-126", "mode": "writeback"}]': finished 2024-03-26T05:35:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[29560]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T05:35:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:35:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:57 smithi164 ceph-mon[33554]: pgmap v990: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:35:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:57 smithi164 ceph-mon[33554]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T05:35:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:35:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:35:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[29560]: pgmap v990: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:35:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[29560]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T05:35:57.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:57.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[38106]: pgmap v990: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:35:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:35:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[38106]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T05:35:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]: dispatch 2024-03-26T05:35:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:58 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:58 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:58.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]': finished 2024-03-26T05:35:58.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:58 smithi164 ceph-mon[33554]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T05:35:58.990 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:35:58.990 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9057 ms) 2024-03-26T05:35:58.990 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6026 ms) 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5054 ms) 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26091 ms) 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13223 ms) 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12098 ms) 2024-03-26T05:35:58.991 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: 00000012 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12060 ms) 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6043 ms) 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (555538 ms total) 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1015 ms) 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-26T05:35:58.992 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11040 ms) 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18069 ms) 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30127 ms total) 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7038 ms) 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8068 ms) 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: waiting for scrub... 2024-03-26T05:35:58.993 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: done waiting 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24306 ms) 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9104 ms) 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7972 ms) 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8038 ms) 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10070 ms) 2024-03-26T05:35:58.994 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-26T05:35:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]': finished 2024-03-26T05:35:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[38106]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T05:35:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:35:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:35:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:35:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-126"}]': finished 2024-03-26T05:35:59.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:58 smithi136 ceph-mon[29560]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T05:35:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:59 smithi164 ceph-mon[33554]: pgmap v993: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:35:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:59 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:35:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:35:59 smithi164 ceph-mon[33554]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T05:36:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:59 smithi136 ceph-mon[38106]: pgmap v993: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:59 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:35:59 smithi136 ceph-mon[38106]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T05:36:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:59 smithi136 ceph-mon[29560]: pgmap v993: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:59 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:35:59 smithi136 ceph-mon[29560]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T05:36:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:00 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:36:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:00 smithi164 ceph-mon[33554]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T05:36:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:00 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:36:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:00 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:36:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:00 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:36:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[38106]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T05:36:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:36:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:36:01.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:01.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:36:01.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[29560]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T05:36:01.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:01.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:01.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:36:01.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:36:01.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:00 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:01 smithi164 ceph-mon[33554]: pgmap v996: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:01 smithi164 ceph-mon[33554]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T05:36:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:01.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[38106]: pgmap v996: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[38106]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T05:36:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[29560]: pgmap v996: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[29560]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T05:36:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:02.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:36:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:36:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[38106]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T05:36:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[38106]: pgmap v999: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:03.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:03.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[29560]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T05:36:03.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:03.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:03.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[29560]: pgmap v999: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:03.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:03 smithi164 ceph-mon[33554]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T05:36:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:03 smithi164 ceph-mon[33554]: pgmap v999: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]': finished 2024-03-26T05:36:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:04 smithi136 ceph-mon[38106]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T05:36:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]: dispatch 2024-03-26T05:36:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]: dispatch 2024-03-26T05:36:04.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]': finished 2024-03-26T05:36:04.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:04 smithi136 ceph-mon[29560]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T05:36:04.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]: dispatch 2024-03-26T05:36:04.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]: dispatch 2024-03-26T05:36:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-128"}]': finished 2024-03-26T05:36:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:04 smithi164 ceph-mon[33554]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T05:36:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]: dispatch 2024-03-26T05:36:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]: dispatch 2024-03-26T05:36:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:05 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:05 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]': finished 2024-03-26T05:36:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:05 smithi136 ceph-mon[38106]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T05:36:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:05 smithi136 ceph-mon[38106]: pgmap v1002: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:05 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:05.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:05 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]': finished 2024-03-26T05:36:05.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:05 smithi136 ceph-mon[29560]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T05:36:05.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:05 smithi136 ceph-mon[29560]: pgmap v1002: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:05 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:05 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-128", "mode": "writeback"}]': finished 2024-03-26T05:36:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:05 smithi164 ceph-mon[33554]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T05:36:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:05 smithi164 ceph-mon[33554]: pgmap v1002: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:07 smithi164 ceph-mon[33554]: pgmap v1003: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 403 B/s wr, 1 op/s 2024-03-26T05:36:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:07 smithi136 ceph-mon[38106]: pgmap v1003: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 403 B/s wr, 1 op/s 2024-03-26T05:36:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:36:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:36:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:36:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:07 smithi136 ceph-mon[29560]: pgmap v1003: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 403 B/s wr, 1 op/s 2024-03-26T05:36:09.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:09 smithi164 ceph-mon[33554]: pgmap v1004: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 336 B/s wr, 1 op/s 2024-03-26T05:36:09.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:09 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:09.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:09.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:09.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:09 smithi136 ceph-mon[38106]: pgmap v1004: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 336 B/s wr, 1 op/s 2024-03-26T05:36:09.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:09 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:09.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:09.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:09.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:09 smithi136 ceph-mon[29560]: pgmap v1004: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 336 B/s wr, 1 op/s 2024-03-26T05:36:09.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:09 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:09.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:09.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:10 smithi164 ceph-mon[33554]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T05:36:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:10.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:10 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:10.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]': finished 2024-03-26T05:36:10.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:10 smithi164 ceph-mon[33554]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T05:36:10.476 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:10.477 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[38106]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T05:36:10.477 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:10.477 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:10.477 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:10.477 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]': finished 2024-03-26T05:36:10.477 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[38106]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T05:36:10.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:10.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[29560]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T05:36:10.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:10.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]: dispatch 2024-03-26T05:36:10.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:10.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-128"}]': finished 2024-03-26T05:36:10.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:10 smithi136 ceph-mon[29560]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T05:36:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:11 smithi164 ceph-mon[33554]: pgmap v1006: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 289 B/s wr, 1 op/s 2024-03-26T05:36:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:11 smithi136 ceph-mon[38106]: pgmap v1006: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 289 B/s wr, 1 op/s 2024-03-26T05:36:11.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:11 smithi136 ceph-mon[29560]: pgmap v1006: 236 pgs: 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 289 B/s wr, 1 op/s 2024-03-26T05:36:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:12 smithi164 ceph-mon[33554]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T05:36:12.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:12 smithi136 ceph-mon[38106]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T05:36:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:12 smithi136 ceph-mon[29560]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T05:36:12.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:36:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:36:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:13 smithi164 ceph-mon[33554]: pgmap v1009: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:36:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:13 smithi164 ceph-mon[33554]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T05:36:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:13.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:13.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:13.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:13 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:13.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[38106]: pgmap v1009: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:36:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[38106]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T05:36:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[29560]: pgmap v1009: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T05:36:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[29560]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T05:36:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:14 smithi164 ceph-mon[33554]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T05:36:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:14.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:14 smithi164 ceph-mon[33554]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T05:36:14.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:14.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[38106]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T05:36:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[38106]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T05:36:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:14.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:14.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:14.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[29560]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T05:36:14.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:14.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:14.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:14.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[29560]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T05:36:14.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:14.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:15 smithi164 ceph-mon[33554]: pgmap v1012: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:36:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]': finished 2024-03-26T05:36:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:15 smithi164 ceph-mon[33554]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T05:36:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]: dispatch 2024-03-26T05:36:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]: dispatch 2024-03-26T05:36:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[38106]: pgmap v1012: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:36:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]': finished 2024-03-26T05:36:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[38106]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T05:36:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]: dispatch 2024-03-26T05:36:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]: dispatch 2024-03-26T05:36:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[29560]: pgmap v1012: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:36:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-130"}]': finished 2024-03-26T05:36:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[29560]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T05:36:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]: dispatch 2024-03-26T05:36:15.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]: dispatch 2024-03-26T05:36:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:16 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]': finished 2024-03-26T05:36:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:16 smithi164 ceph-mon[33554]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T05:36:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:16 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]': finished 2024-03-26T05:36:16.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:16 smithi136 ceph-mon[38106]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T05:36:16.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:16 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-130", "mode": "writeback"}]': finished 2024-03-26T05:36:16.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:16 smithi136 ceph-mon[29560]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T05:36:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:17 smithi164 ceph-mon[33554]: pgmap v1015: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:17.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:36:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:36:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:36:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:17 smithi136 ceph-mon[38106]: pgmap v1015: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:17 smithi136 ceph-mon[29560]: pgmap v1015: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:18.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:18.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:18 smithi164 ceph-mon[33554]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T05:36:18.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:18.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:18.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:18.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:18 smithi136 ceph-mon[38106]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T05:36:18.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:18.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:18.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:18.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:18 smithi136 ceph-mon[29560]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T05:36:18.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:18.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]: dispatch 2024-03-26T05:36:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:19 smithi164 ceph-mon[33554]: pgmap v1018: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:19 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]': finished 2024-03-26T05:36:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:19 smithi164 ceph-mon[33554]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T05:36:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:19 smithi136 ceph-mon[29560]: pgmap v1018: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:19 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]': finished 2024-03-26T05:36:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:19 smithi136 ceph-mon[29560]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T05:36:19.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:19 smithi136 ceph-mon[38106]: pgmap v1018: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:19.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:19 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:19.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-130"}]': finished 2024-03-26T05:36:19.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:19 smithi136 ceph-mon[38106]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T05:36:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:20 smithi164 ceph-mon[33554]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T05:36:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:20 smithi164 ceph-mon[33554]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T05:36:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:20.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:20 smithi136 ceph-mon[38106]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T05:36:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:20 smithi136 ceph-mon[38106]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T05:36:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:20.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:20.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:20 smithi136 ceph-mon[29560]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T05:36:20.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:20 smithi136 ceph-mon[29560]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T05:36:20.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:20.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[38106]: pgmap v1021: 204 pgs: 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[38106]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T05:36:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:21.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[29560]: pgmap v1021: 204 pgs: 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:21.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:21.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[29560]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T05:36:21.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:21.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:21 smithi164 ceph-mon[33554]: pgmap v1021: 204 pgs: 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:21 smithi164 ceph-mon[33554]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T05:36:21.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:21.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:22.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:36:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:36:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[29560]: pgmap v1024: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[29560]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T05:36:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:23.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:23.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[38106]: pgmap v1024: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:23.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:23.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[38106]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T05:36:23.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:23.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:23.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:23 smithi164 ceph-mon[33554]: pgmap v1024: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:23 smithi164 ceph-mon[33554]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T05:36:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:23.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[38106]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T05:36:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]: dispatch 2024-03-26T05:36:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]: dispatch 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]': finished 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[38106]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[29560]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]: dispatch 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]: dispatch 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]': finished 2024-03-26T05:36:24.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:24 smithi136 ceph-mon[29560]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T05:36:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:24 smithi164 ceph-mon[33554]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T05:36:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]: dispatch 2024-03-26T05:36:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]: dispatch 2024-03-26T05:36:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:24 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:24.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-132", "mode": "writeback"}]': finished 2024-03-26T05:36:24.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:24 smithi164 ceph-mon[33554]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T05:36:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:25 smithi136 ceph-mon[38106]: pgmap v1027: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:25.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:25 smithi136 ceph-mon[29560]: pgmap v1027: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:25 smithi164 ceph-mon[33554]: pgmap v1027: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:36:26.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:26 smithi136 ceph-mon[29560]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T05:36:26.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:26 smithi136 ceph-mon[38106]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T05:36:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:26 smithi164 ceph-mon[33554]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T05:36:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:36:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:36:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-mon[29560]: pgmap v1030: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:36:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-mon[29560]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T05:36:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-mon[38106]: pgmap v1030: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:36:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-mon[38106]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T05:36:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:27 smithi164 ceph-mon[33554]: pgmap v1030: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:36:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:27 smithi164 ceph-mon[33554]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T05:36:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[29560]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T05:36:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:28.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:28.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[38106]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T05:36:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:28 smithi164 ceph-mon[33554]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T05:36:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:28.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:28.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:28 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:28.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:29.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:29 smithi136 ceph-mon[29560]: pgmap v1033: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:36:29.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:29.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:29 smithi136 ceph-mon[29560]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T05:36:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:29 smithi136 ceph-mon[38106]: pgmap v1033: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:36:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:29.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:29 smithi136 ceph-mon[38106]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T05:36:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:29 smithi164 ceph-mon[33554]: pgmap v1033: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:36:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:29 smithi164 ceph-mon[33554]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T05:36:30.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:30 smithi136 ceph-mon[38106]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T05:36:30.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:30.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:30.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:30 smithi136 ceph-mon[29560]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T05:36:30.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:30.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:30.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:30 smithi164 ceph-mon[33554]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T05:36:30.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:30.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[38106]: pgmap v1036: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:31.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[38106]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T05:36:31.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:31.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[29560]: pgmap v1036: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:31.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[29560]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T05:36:31.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:31.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:31 smithi164 ceph-mon[33554]: pgmap v1036: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:31 smithi164 ceph-mon[33554]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T05:36:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]: dispatch 2024-03-26T05:36:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:32 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:32 smithi136 ceph-mon[38106]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T05:36:32.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:32 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:32.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:32 smithi136 ceph-mon[29560]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T05:36:32.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:32 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:32.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-132"}]': finished 2024-03-26T05:36:32.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:32 smithi164 ceph-mon[33554]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T05:36:32.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:36:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:36:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:33 smithi136 ceph-mon[38106]: pgmap v1039: 236 pgs: 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:36:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:33 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:33 smithi136 ceph-mon[38106]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T05:36:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:33 smithi136 ceph-mon[29560]: pgmap v1039: 236 pgs: 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:36:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:33 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:33 smithi136 ceph-mon[29560]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T05:36:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:33 smithi164 ceph-mon[33554]: pgmap v1039: 236 pgs: 236 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:36:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:33 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:33 smithi164 ceph-mon[33554]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T05:36:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[38106]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T05:36:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[38106]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[29560]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[29560]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:34 smithi164 ceph-mon[33554]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T05:36:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:34.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:34 smithi164 ceph-mon[33554]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T05:36:34.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:34.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[38106]: pgmap v1043: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 269 B/s wr, 2 op/s 2024-03-26T05:36:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[38106]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T05:36:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[29560]: pgmap v1043: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 269 B/s wr, 2 op/s 2024-03-26T05:36:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[29560]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T05:36:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:35.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:35 smithi164 ceph-mon[33554]: pgmap v1043: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 269 B/s wr, 2 op/s 2024-03-26T05:36:35.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:35.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:35 smithi164 ceph-mon[33554]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T05:36:35.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:35.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:37.325 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:36:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:36:37.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]': finished 2024-03-26T05:36:37.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[38106]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T05:36:37.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]: dispatch 2024-03-26T05:36:37.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]: dispatch 2024-03-26T05:36:37.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[38106]: pgmap v1046: 236 pgs: 236 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:36:37.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]': finished 2024-03-26T05:36:37.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[29560]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T05:36:37.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]: dispatch 2024-03-26T05:36:37.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]: dispatch 2024-03-26T05:36:37.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:36 smithi136 ceph-mon[29560]: pgmap v1046: 236 pgs: 236 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:36:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-134"}]': finished 2024-03-26T05:36:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:36 smithi164 ceph-mon[33554]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T05:36:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]: dispatch 2024-03-26T05:36:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]: dispatch 2024-03-26T05:36:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:36 smithi164 ceph-mon[33554]: pgmap v1046: 236 pgs: 236 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:36:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]': finished 2024-03-26T05:36:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[38106]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T05:36:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]': finished 2024-03-26T05:36:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[29560]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T05:36:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:37 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-134", "mode": "writeback"}]': finished 2024-03-26T05:36:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:37 smithi164 ceph-mon[33554]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T05:36:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[38106]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T05:36:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[38106]: pgmap v1049: 236 pgs: 236 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[29560]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T05:36:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[29560]: pgmap v1049: 236 pgs: 236 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:39.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:39 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:39 smithi164 ceph-mon[33554]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T05:36:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]: dispatch 2024-03-26T05:36:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:39 smithi164 ceph-mon[33554]: pgmap v1049: 236 pgs: 236 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:39 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:40 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:40 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]': finished 2024-03-26T05:36:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:40 smithi136 ceph-mon[38106]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T05:36:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:40 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:40 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]': finished 2024-03-26T05:36:40.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:40 smithi136 ceph-mon[29560]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T05:36:40.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:40 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:40.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:40 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-134"}]': finished 2024-03-26T05:36:40.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:40 smithi164 ceph-mon[33554]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T05:36:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:41 smithi136 ceph-mon[38106]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T05:36:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:41 smithi136 ceph-mon[38106]: pgmap v1052: 204 pgs: 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:41.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:41 smithi136 ceph-mon[29560]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T05:36:41.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:41 smithi136 ceph-mon[29560]: pgmap v1052: 204 pgs: 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:41 smithi164 ceph-mon[33554]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T05:36:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:41 smithi164 ceph-mon[33554]: pgmap v1052: 204 pgs: 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:42 smithi136 ceph-mon[38106]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T05:36:42.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:42 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:42.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:42.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:42 smithi136 ceph-mon[29560]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T05:36:42.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:42 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:42.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:42 smithi164 ceph-mon[33554]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T05:36:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:42 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:42.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:36:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:36:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T05:36:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: pgmap v1055: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:43.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: pgmap v1055: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:43.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:43 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T05:36:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: pgmap v1055: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:36:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:36:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T05:36:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:43.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:43 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]': finished 2024-03-26T05:36:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[38106]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T05:36:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]: dispatch 2024-03-26T05:36:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]: dispatch 2024-03-26T05:36:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[38106]: pgmap v1058: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:36:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]': finished 2024-03-26T05:36:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[29560]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T05:36:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]: dispatch 2024-03-26T05:36:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]: dispatch 2024-03-26T05:36:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:45 smithi136 ceph-mon[29560]: pgmap v1058: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:36:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-136"}]': finished 2024-03-26T05:36:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:45 smithi164 ceph-mon[33554]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T05:36:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:45 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]: dispatch 2024-03-26T05:36:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:45 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]: dispatch 2024-03-26T05:36:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:45 smithi164 ceph-mon[33554]: pgmap v1058: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:36:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]': finished 2024-03-26T05:36:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[38106]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T05:36:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]': finished 2024-03-26T05:36:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[29560]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T05:36:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:46 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:46 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:36:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-136", "mode": "writeback"}]': finished 2024-03-26T05:36:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:46 smithi164 ceph-mon[33554]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T05:36:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:46 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:46 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:47.325 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:36:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:36:47.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[38106]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[38106]: pgmap v1061: 236 pgs: 236 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[29560]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:47.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:47 smithi136 ceph-mon[29560]: pgmap v1061: 236 pgs: 236 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T05:36:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:36:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:47 smithi164 ceph-mon[33554]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T05:36:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]: dispatch 2024-03-26T05:36:47.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:47 smithi164 ceph-mon[33554]: pgmap v1061: 236 pgs: 236 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T05:36:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:48 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]': finished 2024-03-26T05:36:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:48 smithi136 ceph-mon[38106]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T05:36:48.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:48 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:48.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]': finished 2024-03-26T05:36:48.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:48 smithi136 ceph-mon[29560]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T05:36:48.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:48 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:36:48.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-136"}]': finished 2024-03-26T05:36:48.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:48 smithi164 ceph-mon[33554]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T05:36:49.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:49 smithi136 ceph-mon[38106]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T05:36:49.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:49 smithi136 ceph-mon[38106]: pgmap v1064: 204 pgs: 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T05:36:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:49 smithi136 ceph-mon[29560]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T05:36:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:49 smithi136 ceph-mon[29560]: pgmap v1064: 204 pgs: 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T05:36:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:49 smithi164 ceph-mon[33554]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T05:36:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:49 smithi164 ceph-mon[33554]: pgmap v1064: 204 pgs: 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T05:36:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:50 smithi136 ceph-mon[38106]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T05:36:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:50 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:50.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:50 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:50 smithi136 ceph-mon[29560]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T05:36:50.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:50 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:50.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:50 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:50 smithi164 ceph-mon[33554]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T05:36:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:50 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:50 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:51.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:51.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T05:36:51.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:51.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:51.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: pgmap v1067: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:51.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:51.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T05:36:51.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:36:51.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:36:51.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:51.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T05:36:51.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:51.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:51.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: pgmap v1067: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:51.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:51.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T05:36:51.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:36:51.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:36:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:36:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T05:36:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:36:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: pgmap v1067: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:36:51.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:36:51.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T05:36:51.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:36:51.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:36:52.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:36:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:36:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:36:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[38106]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T05:36:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:36:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:36:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[38106]: pgmap v1070: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:53.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:36:53.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[29560]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T05:36:53.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:36:53.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:36:53.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[29560]: pgmap v1070: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:53.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:36:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:53 smithi164 ceph-mon[33554]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T05:36:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:36:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:36:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:53 smithi164 ceph-mon[33554]: pgmap v1070: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:36:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:36:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:54 smithi136 ceph-mon[38106]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T05:36:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:36:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:36:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:36:54.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:54 smithi136 ceph-mon[29560]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T05:36:54.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:36:54.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:36:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:36:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:54 smithi164 ceph-mon[33554]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T05:36:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:36:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:36:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:36:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:55 smithi164 ceph-mon[33554]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T05:36:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:55 smithi164 ceph-mon[33554]: pgmap v1073: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:36:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:55 smithi136 ceph-mon[38106]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T05:36:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:55 smithi136 ceph-mon[38106]: pgmap v1073: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:36:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:55 smithi136 ceph-mon[29560]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T05:36:55.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:55 smithi136 ceph-mon[29560]: pgmap v1073: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]: dispatch 2024-03-26T05:36:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]: dispatch 2024-03-26T05:36:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]: dispatch 2024-03-26T05:36:56.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]: dispatch 2024-03-26T05:36:56.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:36:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]: dispatch 2024-03-26T05:36:56.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]: dispatch 2024-03-26T05:36:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]': finished 2024-03-26T05:36:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:57 smithi164 ceph-mon[33554]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T05:36:57.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:57 smithi164 ceph-mon[33554]: pgmap v1075: 236 pgs: 236 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]': finished 2024-03-26T05:36:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:57 smithi136 ceph-mon[29560]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T05:36:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:57 smithi136 ceph-mon[29560]: pgmap v1075: 236 pgs: 236 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:36:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:36:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:36:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-138"}]': finished 2024-03-26T05:36:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:57 smithi136 ceph-mon[38106]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T05:36:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:57 smithi136 ceph-mon[38106]: pgmap v1075: 236 pgs: 236 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:36:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:58 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:58 smithi164 ceph-mon[33554]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T05:36:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:58.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:58 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:58.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:58 smithi136 ceph-mon[29560]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T05:36:58.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:58 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:36:58.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:58 smithi136 ceph-mon[38106]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T05:36:58.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:36:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:59 smithi164 ceph-mon[33554]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-26T05:36:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:59 smithi164 ceph-mon[33554]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T05:36:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:59 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:36:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:59 smithi136 ceph-mon[29560]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-26T05:36:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:59 smithi136 ceph-mon[29560]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T05:36:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:59 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:36:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:59.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:59 smithi136 ceph-mon[38106]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-26T05:36:59.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:59 smithi136 ceph-mon[38106]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T05:36:59.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:59 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:36:59.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:36:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:00.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:00.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:00 smithi164 ceph-mon[33554]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T05:37:00.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:00 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:00.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:00 smithi136 ceph-mon[29560]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T05:37:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:00 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:00.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:00.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:00.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:00 smithi136 ceph-mon[38106]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T05:37:00.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:00 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:00.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:01 smithi164 ceph-mon[33554]: pgmap v1080: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:37:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:01 smithi164 ceph-mon[33554]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T05:37:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:37:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:37:01.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:01 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:37:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[38106]: pgmap v1080: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:37:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[38106]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T05:37:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:37:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:37:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:37:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[29560]: pgmap v1080: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:37:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[29560]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T05:37:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:37:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:37:01.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:01 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:37:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:37:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T05:37:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:37:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:37:02.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:37:02.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:37:02.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:37:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T05:37:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:37:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:37:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:37:02.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:37:02.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:02.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:37:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:37:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:03 smithi164 ceph-mon[33554]: pgmap v1083: 236 pgs: 236 active+clean; 4.4 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:37:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:03 smithi164 ceph-mon[33554]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T05:37:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:03.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:03 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:03.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[38106]: pgmap v1083: 236 pgs: 236 active+clean; 4.4 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:37:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[38106]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T05:37:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[29560]: pgmap v1083: 236 pgs: 236 active+clean; 4.4 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:37:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[29560]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T05:37:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:03.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:04 smithi164 ceph-mon[33554]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T05:37:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:37:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:37:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:04 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:04 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:37:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[38106]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T05:37:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:37:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:37:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:37:04.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:04.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[29560]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T05:37:04.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:37:04.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:37:04.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:04.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:04 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:37:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:05 smithi164 ceph-mon[33554]: pgmap v1086: 236 pgs: 236 active+clean; 4.4 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:05 smithi164 ceph-mon[33554]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T05:37:05.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:05 smithi136 ceph-mon[38106]: pgmap v1086: 236 pgs: 236 active+clean; 4.4 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:05.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:05 smithi136 ceph-mon[38106]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T05:37:05.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:05 smithi136 ceph-mon[29560]: pgmap v1086: 236 pgs: 236 active+clean; 4.4 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:05.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:05 smithi136 ceph-mon[29560]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T05:37:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:07 smithi164 ceph-mon[33554]: pgmap v1088: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 3.2 KiB/s wr, 1 op/s 2024-03-26T05:37:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:07 smithi136 ceph-mon[29560]: pgmap v1088: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 3.2 KiB/s wr, 1 op/s 2024-03-26T05:37:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:07 smithi136 ceph-mon[38106]: pgmap v1088: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 3.2 KiB/s wr, 1 op/s 2024-03-26T05:37:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:37:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:37:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:37:09.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:09 smithi164 ceph-mon[33554]: pgmap v1089: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-26T05:37:09.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:09 smithi136 ceph-mon[29560]: pgmap v1089: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-26T05:37:09.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:09 smithi136 ceph-mon[38106]: pgmap v1089: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-26T05:37:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:11 smithi164 ceph-mon[33554]: pgmap v1090: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 643 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T05:37:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:11 smithi136 ceph-mon[38106]: pgmap v1090: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 643 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T05:37:11.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:11 smithi136 ceph-mon[29560]: pgmap v1090: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 643 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T05:37:12.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:37:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:37:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:13 smithi164 ceph-mon[33554]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-03-26T05:37:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:13 smithi136 ceph-mon[38106]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-03-26T05:37:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:13 smithi136 ceph-mon[29560]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-03-26T05:37:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:15 smithi164 ceph-mon[33554]: pgmap v1092: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T05:37:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:15 smithi136 ceph-mon[38106]: pgmap v1092: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T05:37:15.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:15 smithi136 ceph-mon[29560]: pgmap v1092: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T05:37:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:17 smithi164 ceph-mon[33554]: pgmap v1093: 236 pgs: 236 active+clean; 4.5 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T05:37:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]: dispatch 2024-03-26T05:37:17.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]: dispatch 2024-03-26T05:37:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:37:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:37:17.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[29560]: pgmap v1093: 236 pgs: 236 active+clean; 4.5 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]: dispatch 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]: dispatch 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[38106]: pgmap v1093: 236 pgs: 236 active+clean; 4.5 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]: dispatch 2024-03-26T05:37:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]: dispatch 2024-03-26T05:37:18.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]': finished 2024-03-26T05:37:18.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:18 smithi164 ceph-mon[33554]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T05:37:18.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]': finished 2024-03-26T05:37:18.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:18 smithi136 ceph-mon[38106]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T05:37:18.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-140"}]': finished 2024-03-26T05:37:18.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:18 smithi136 ceph-mon[29560]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T05:37:19.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:19 smithi164 ceph-mon[33554]: pgmap v1095: 236 pgs: 236 active+clean; 4.5 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T05:37:19.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:19 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:19.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:19 smithi164 ceph-mon[33554]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T05:37:19.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:19 smithi136 ceph-mon[38106]: pgmap v1095: 236 pgs: 236 active+clean; 4.5 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T05:37:19.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:19 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:19.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:19 smithi136 ceph-mon[38106]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T05:37:19.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:19 smithi136 ceph-mon[29560]: pgmap v1095: 236 pgs: 236 active+clean; 4.5 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T05:37:19.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:19 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:19.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:19 smithi136 ceph-mon[29560]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T05:37:20.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:20 smithi164 ceph-mon[33554]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T05:37:20.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:20.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:20.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:20 smithi136 ceph-mon[38106]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T05:37:20.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:20.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:20 smithi136 ceph-mon[29560]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T05:37:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:21 smithi164 ceph-mon[33554]: pgmap v1098: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:37:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:21 smithi164 ceph-mon[33554]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T05:37:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[38106]: pgmap v1098: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:37:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[38106]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T05:37:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[29560]: pgmap v1098: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:37:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[29560]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T05:37:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:21.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:22.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:22.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:22.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:22 smithi136 ceph-mon[38106]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T05:37:22.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:22.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:22.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:22.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:22 smithi136 ceph-mon[29560]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T05:37:22.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:22.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:22.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:22.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:22 smithi164 ceph-mon[33554]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T05:37:22.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:22.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:37:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:37:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[38106]: pgmap v1101: 236 pgs: 236 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:37:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]': finished 2024-03-26T05:37:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[38106]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T05:37:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]: dispatch 2024-03-26T05:37:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]: dispatch 2024-03-26T05:37:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[29560]: pgmap v1101: 236 pgs: 236 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:37:23.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]': finished 2024-03-26T05:37:23.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[29560]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T05:37:23.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]: dispatch 2024-03-26T05:37:23.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]: dispatch 2024-03-26T05:37:23.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:23 smithi164 ceph-mon[33554]: pgmap v1101: 236 pgs: 236 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T05:37:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-142"}]': finished 2024-03-26T05:37:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:23 smithi164 ceph-mon[33554]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T05:37:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]: dispatch 2024-03-26T05:37:23.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]: dispatch 2024-03-26T05:37:23.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]': finished 2024-03-26T05:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[38106]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T05:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]': finished 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[29560]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:24.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:37:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:24 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-142", "mode": "writeback"}]': finished 2024-03-26T05:37:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:24 smithi164 ceph-mon[33554]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T05:37:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:24.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:24.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:24 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:24.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:37:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[38106]: pgmap v1104: 236 pgs: 236 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:37:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[38106]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T05:37:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:37:25.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[29560]: pgmap v1104: 236 pgs: 236 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:37:25.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[29560]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T05:37:25.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:25.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:25.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:37:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:25 smithi164 ceph-mon[33554]: pgmap v1104: 236 pgs: 236 active+clean; 4.4 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:37:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:25 smithi164 ceph-mon[33554]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T05:37:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:37:26.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[38106]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T05:37:26.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:26.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:26.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:26.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:26.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[38106]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T05:37:26.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:26.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[29560]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T05:37:26.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:26.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:26.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:26.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:26.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[29560]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T05:37:26.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:26 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:26 smithi164 ceph-mon[33554]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T05:37:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:26 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:26 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:26 smithi164 ceph-mon[33554]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T05:37:26.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:26 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:37:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:37:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[38106]: pgmap v1107: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:37:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[38106]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T05:37:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:37:27.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:37:27.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[29560]: pgmap v1107: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:27.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:27.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:37:27.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[29560]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T05:37:27.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:37:27.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:37:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:27 smithi164 ceph-mon[33554]: pgmap v1107: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:37:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:27 smithi164 ceph-mon[33554]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T05:37:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:37:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:37:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:37:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[29560]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T05:37:28.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:37:28.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:37:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:37:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[38106]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T05:37:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:37:28.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:37:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:37:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:28 smithi164 ceph-mon[33554]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T05:37:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:37:28.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:37:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:29 smithi136 ceph-mon[38106]: pgmap v1110: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:37:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:29 smithi136 ceph-mon[38106]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T05:37:29.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:29 smithi136 ceph-mon[29560]: pgmap v1110: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:29.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:37:29.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:29 smithi136 ceph-mon[29560]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T05:37:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:29 smithi164 ceph-mon[33554]: pgmap v1110: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:37:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:29 smithi164 ceph-mon[33554]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T05:37:30.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:30 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:30.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:30 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:30.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:30 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:30.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:30 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:30.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:30 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:30.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:30 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[38106]: pgmap v1113: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:37:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:37:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[38106]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T05:37:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[29560]: pgmap v1113: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:37:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:37:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[29560]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T05:37:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:31.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:31 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:31 smithi164 ceph-mon[33554]: pgmap v1113: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:37:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:37:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:31 smithi164 ceph-mon[33554]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T05:37:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:31 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:31 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]': finished 2024-03-26T05:37:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[38106]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T05:37:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:32.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]': finished 2024-03-26T05:37:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[29560]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T05:37:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:32.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:32 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:32.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]': finished 2024-03-26T05:37:32.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:32 smithi164 ceph-mon[33554]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T05:37:32.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:32.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:32.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:32 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:32.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:32 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-142"}]: dispatch 2024-03-26T05:37:32.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:37:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:37:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:33 smithi136 ceph-mon[38106]: pgmap v1116: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:37:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:33 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:33 smithi136 ceph-mon[38106]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T05:37:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:33.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:33 smithi136 ceph-mon[29560]: pgmap v1116: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:37:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:33 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:33 smithi136 ceph-mon[29560]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T05:37:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:33 smithi164 ceph-mon[33554]: pgmap v1116: 236 pgs: 236 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:37:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:33 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:33 smithi164 ceph-mon[33554]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T05:37:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:34 smithi136 ceph-mon[38106]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T05:37:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:34 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:34.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:34 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:34.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:34 smithi136 ceph-mon[29560]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T05:37:34.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:34 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:34.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:34 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:34 smithi164 ceph-mon[33554]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T05:37:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:34 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:34 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[38106]: pgmap v1119: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:37:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[38106]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T05:37:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[29560]: pgmap v1119: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:37:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[29560]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T05:37:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:35.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:35 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:35.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:35 smithi164 ceph-mon[33554]: pgmap v1119: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 708 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:37:35.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:35.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:35 smithi164 ceph-mon[33554]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T05:37:35.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:35.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:35 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:36.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:36.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:36 smithi136 ceph-mon[38106]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T05:37:36.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:36.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:36 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:36.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:36.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:36 smithi136 ceph-mon[29560]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T05:37:36.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:36.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:36 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:36 smithi164 ceph-mon[33554]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T05:37:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:36.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:36 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:37:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:37:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[29560]: pgmap v1122: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:37:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]': finished 2024-03-26T05:37:37.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[29560]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T05:37:37.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:37:37.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:37:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[38106]: pgmap v1122: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:37:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]': finished 2024-03-26T05:37:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[38106]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T05:37:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:37:37.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:37 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:37:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:37 smithi164 ceph-mon[33554]: pgmap v1122: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:37:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-144"}]': finished 2024-03-26T05:37:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:37 smithi164 ceph-mon[33554]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T05:37:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:37:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:37 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:37:38.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:38 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:38.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:38 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]': finished 2024-03-26T05:37:38.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:38 smithi136 ceph-mon[38106]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T05:37:38.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:38 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:38.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:38 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]': finished 2024-03-26T05:37:38.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:38 smithi136 ceph-mon[29560]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T05:37:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:38 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:38 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-144", "mode": "readproxy"}]': finished 2024-03-26T05:37:38.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:38 smithi164 ceph-mon[33554]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T05:37:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:39 smithi164 ceph-mon[33554]: pgmap v1125: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:37:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:39 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:39 smithi136 ceph-mon[38106]: pgmap v1125: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:37:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:39 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:39 smithi136 ceph-mon[29560]: pgmap v1125: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:37:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:39 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:37:41.627 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:41 smithi136 ceph-mon[29560]: pgmap v1126: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 176 B/s wr, 1 op/s 2024-03-26T05:37:41.628 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:41 smithi136 ceph-mon[38106]: pgmap v1126: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 176 B/s wr, 1 op/s 2024-03-26T05:37:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:41 smithi164 ceph-mon[33554]: pgmap v1126: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 176 B/s wr, 1 op/s 2024-03-26T05:37:42.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:37:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:37:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:43 smithi164 ceph-mon[33554]: pgmap v1127: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 150 B/s wr, 1 op/s 2024-03-26T05:37:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:43 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:43 smithi136 ceph-mon[29560]: pgmap v1127: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 150 B/s wr, 1 op/s 2024-03-26T05:37:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:43 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:43 smithi136 ceph-mon[38106]: pgmap v1127: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 150 B/s wr, 1 op/s 2024-03-26T05:37:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:43 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:37:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:45 smithi164 ceph-mon[33554]: pgmap v1128: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:37:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:45 smithi136 ceph-mon[29560]: pgmap v1128: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:37:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:45 smithi136 ceph-mon[38106]: pgmap v1128: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:37:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:37:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:37:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:37:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:47 smithi164 ceph-mon[33554]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:37:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:47 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:47.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:47 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:47 smithi136 ceph-mon[29560]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:37:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:47 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:47 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:47 smithi136 ceph-mon[38106]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:37:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:47 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:47 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:37:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:48 smithi164 ceph-mon[33554]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T05:37:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:48 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:48.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:48 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:49.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:37:49.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:48 smithi136 ceph-mon[29560]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T05:37:49.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:48 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:49.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:48 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:49.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:37:49.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:48 smithi136 ceph-mon[38106]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T05:37:49.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:48 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:49.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:48 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:49 smithi164 ceph-mon[33554]: pgmap v1131: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:37:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:49 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]': finished 2024-03-26T05:37:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:49 smithi164 ceph-mon[33554]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T05:37:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:49.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:49.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:49 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:49.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:49 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[38106]: pgmap v1131: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:37:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]': finished 2024-03-26T05:37:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[38106]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T05:37:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[29560]: pgmap v1131: 236 pgs: 236 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]': finished 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[29560]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:49 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-144"}]: dispatch 2024-03-26T05:37:51.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:50 smithi136 ceph-mon[38106]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T05:37:51.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:50 smithi136 ceph-mon[29560]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T05:37:51.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:50 smithi164 ceph-mon[33554]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T05:37:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:51 smithi136 ceph-mon[38106]: pgmap v1134: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:37:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:51 smithi136 ceph-mon[38106]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T05:37:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:51 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:51 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:51 smithi136 ceph-mon[29560]: pgmap v1134: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:37:52.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:51 smithi136 ceph-mon[29560]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T05:37:52.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:51 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:52.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:51 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:51 smithi164 ceph-mon[33554]: pgmap v1134: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:37:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:51 smithi164 ceph-mon[33554]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T05:37:52.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:51 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:52.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:51 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:37:52.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:37:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:52.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-mon[33554]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T05:37:52.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:52.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:52.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:52.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-mon[33554]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T05:37:52.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:37:52.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:52 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:37:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[38106]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T05:37:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:53.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[38106]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[29560]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[29560]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:37:53.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:52 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:37:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[38106]: pgmap v1137: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]': finished 2024-03-26T05:37:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[38106]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T05:37:54.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]: dispatch 2024-03-26T05:37:54.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]: dispatch 2024-03-26T05:37:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[29560]: pgmap v1137: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]': finished 2024-03-26T05:37:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[29560]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T05:37:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]: dispatch 2024-03-26T05:37:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:53 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]: dispatch 2024-03-26T05:37:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:53 smithi164 ceph-mon[33554]: pgmap v1137: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:37:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi136-75071-111", "overlaypool": "test-rados-api-smithi136-75071-146"}]': finished 2024-03-26T05:37:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:53 smithi164 ceph-mon[33554]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T05:37:54.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:53 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]: dispatch 2024-03-26T05:37:54.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:53 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]: dispatch 2024-03-26T05:37:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:54 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:54 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]': finished 2024-03-26T05:37:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:54 smithi136 ceph-mon[38106]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T05:37:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:54 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:54 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]': finished 2024-03-26T05:37:55.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:54 smithi136 ceph-mon[29560]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T05:37:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:54 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:37:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:54 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi136-75071-146", "mode": "writeback"}]': finished 2024-03-26T05:37:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:54 smithi164 ceph-mon[33554]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T05:37:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:55 smithi136 ceph-mon[38106]: pgmap v1140: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:55 smithi136 ceph-mon[29560]: pgmap v1140: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:55 smithi164 ceph-mon[33554]: pgmap v1140: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:37:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:37:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:37:57.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:56 smithi136 ceph-mon[38106]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T05:37:57.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:57.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:37:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:56 smithi136 ceph-mon[29560]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T05:37:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:57.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:37:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:56 smithi164 ceph-mon[33554]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T05:37:57.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:57.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:37:57.577 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:37:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:37:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[29560]: pgmap v1143: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:37:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:37:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[29560]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T05:37:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[38106]: pgmap v1143: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:37:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:37:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[38106]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T05:37:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:58.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:57 smithi164 ceph-mon[33554]: pgmap v1143: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:37:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:37:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:57 smithi164 ceph-mon[33554]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T05:37:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:57 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:57 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:37:58.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:37:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[29560]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T05:37:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:59.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:59.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:59.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[38106]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T05:37:59.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:59.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:58 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:58 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:37:59.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:37:59.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:58 smithi164 ceph-mon[33554]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T05:37:59.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:58 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:37:59.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:58 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:38:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[29560]: pgmap v1146: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:38:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:38:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[29560]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T05:38:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:38:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:38:00.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[38106]: pgmap v1146: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:38:00.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:38:00.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[38106]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T05:38:00.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:38:00.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:37:59 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:38:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:59 smithi164 ceph-mon[33554]: pgmap v1146: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T05:38:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:38:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:59 smithi164 ceph-mon[33554]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T05:38:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:59 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:38:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:37:59 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:38:01.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:00 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:38:01.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:00 smithi136 ceph-mon[29560]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T05:38:01.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:00 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:38:01.079 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:00 smithi136 ceph-mon[38106]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T05:38:01.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:00 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:38:01.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:00 smithi164 ceph-mon[33554]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T05:38:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:01 smithi136 ceph-mon[29560]: pgmap v1149: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:38:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:01 smithi136 ceph-mon[29560]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:38:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:01 smithi136 ceph-mon[38106]: pgmap v1149: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:38:02.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:01 smithi136 ceph-mon[38106]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:38:02.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:01 smithi164 ceph-mon[33554]: pgmap v1149: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:38:02.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:01 smithi164 ceph-mon[33554]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:38:02.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:38:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:38:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:02 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:38:02.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:02.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:02 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:02.970 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:38:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=sqlstore.transactions t=2024-03-26T05:38:02.722360542Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T05:38:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:02 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:38:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:03.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:02 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:03.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:02 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:38:03.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:03.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:02 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[38106]: pgmap v1150: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:38:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:38:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[29560]: pgmap v1150: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:38:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:38:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:03 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:03 smithi164 ceph-mon[33554]: pgmap v1150: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:03 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:38:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:03 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:38:04.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:03 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:05 smithi136 ceph-mon[38106]: pgmap v1151: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:05 smithi136 ceph-mon[29560]: pgmap v1151: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:05 smithi164 ceph-mon[33554]: pgmap v1151: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:38:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:38:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:38:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:07 smithi164 ceph-mon[33554]: pgmap v1152: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:08.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:07 smithi136 ceph-mon[38106]: pgmap v1152: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:08.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:07 smithi136 ceph-mon[29560]: pgmap v1152: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:09 smithi164 ceph-mon[33554]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:09 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:10.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:09 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:09 smithi136 ceph-mon[38106]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:09 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:09 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:09 smithi136 ceph-mon[29560]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:09 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:10.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:09 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:11.121 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:11.121 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:10 smithi164 ceph-mon[33554]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T05:38:11.121 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:10 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:11.121 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:11.121 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:10 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]': finished 2024-03-26T05:38:11.121 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:10 smithi164 ceph-mon[33554]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[38106]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]': finished 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[38106]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[29560]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T05:38:11.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:11.227 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:11.227 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]': finished 2024-03-26T05:38:11.227 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:10 smithi136 ceph-mon[29560]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T05:38:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:11 smithi164 ceph-mon[33554]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:12.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[38106]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:12.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[29560]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:38:12.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:12.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:12.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:12.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-146"}]: dispatch 2024-03-26T05:38:12.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:38:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:38:12.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:12 smithi164 ceph-mon[33554]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T05:38:12.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:12 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:12.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:12 smithi136 ceph-mon[38106]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T05:38:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:12 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:12 smithi136 ceph-mon[29560]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T05:38:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:12 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:13.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:13 smithi164 ceph-mon[33554]: pgmap v1158: 204 pgs: 204 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T05:38:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:13 smithi164 ceph-mon[33554]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:38:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:13 smithi164 ceph-mon[33554]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T05:38:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[38106]: pgmap v1158: 204 pgs: 204 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T05:38:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[38106]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:38:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[38106]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T05:38:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[29560]: pgmap v1158: 204 pgs: 204 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T05:38:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[29560]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:38:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[29560]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T05:38:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:14.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:14 smithi164 ceph-mon[33554]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T05:38:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:38:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:14 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:38:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:14 smithi136 ceph-mon[38106]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T05:38:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:38:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:14 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:38:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:14 smithi136 ceph-mon[29560]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T05:38:15.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:38:15.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:14 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:38:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:15 smithi164 ceph-mon[33554]: pgmap v1161: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:38:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:38:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:15 smithi164 ceph-mon[33554]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T05:38:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[38106]: pgmap v1161: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:38:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:38:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[38106]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T05:38:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[29560]: pgmap v1161: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:38:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:38:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[29560]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T05:38:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]': finished 2024-03-26T05:38:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:16 smithi164 ceph-mon[33554]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T05:38:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:17.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:17.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:16 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:17.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:16 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:17.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]': finished 2024-03-26T05:38:17.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[38106]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T05:38:17.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:17.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:17.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:17.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:17.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]': finished 2024-03-26T05:38:17.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[29560]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T05:38:17.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:17.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:17.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:17.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:16 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-148"}]: dispatch 2024-03-26T05:38:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:38:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:38:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:38:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:17 smithi164 ceph-mon[33554]: pgmap v1164: 236 pgs: 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:38:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:17 smithi164 ceph-mon[33554]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:17 smithi164 ceph-mon[33554]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T05:38:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:17 smithi136 ceph-mon[38106]: pgmap v1164: 236 pgs: 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:38:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:17 smithi136 ceph-mon[38106]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:17 smithi136 ceph-mon[38106]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T05:38:18.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:17 smithi136 ceph-mon[29560]: pgmap v1164: 236 pgs: 236 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T05:38:18.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:17 smithi136 ceph-mon[29560]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:18.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:17 smithi136 ceph-mon[29560]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T05:38:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:18 smithi164 ceph-mon[33554]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T05:38:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:19.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:18 smithi136 ceph-mon[38106]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T05:38:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:19.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:19.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:18 smithi136 ceph-mon[29560]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T05:38:19.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:19.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:19 smithi164 ceph-mon[33554]: pgmap v1167: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:38:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:19 smithi164 ceph-mon[33554]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T05:38:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-150"}]: dispatch 2024-03-26T05:38:20.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-150"}]: dispatch 2024-03-26T05:38:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[38106]: pgmap v1167: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:38:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[38106]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-150"}]: dispatch 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-150"}]: dispatch 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[29560]: pgmap v1167: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[29560]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-150"}]: dispatch 2024-03-26T05:38:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-150"}]: dispatch 2024-03-26T05:38:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:20 smithi164 ceph-mon[33554]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T05:38:21.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:20 smithi136 ceph-mon[38106]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T05:38:21.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:20 smithi136 ceph-mon[29560]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T05:38:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:21 smithi164 ceph-mon[33554]: pgmap v1170: 204 pgs: 204 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:38:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:21 smithi164 ceph-mon[33554]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T05:38:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:22.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:21 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:21 smithi136 ceph-mon[38106]: pgmap v1170: 204 pgs: 204 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:38:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:21 smithi136 ceph-mon[38106]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T05:38:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:21 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:21 smithi136 ceph-mon[29560]: pgmap v1170: 204 pgs: 204 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:38:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:21 smithi136 ceph-mon[29560]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T05:38:22.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:22.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:21 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:22.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:38:23.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:23.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-mon[33554]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T05:38:23.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:23.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:23.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-152"}]: dispatch 2024-03-26T05:38:23.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-152"}]: dispatch 2024-03-26T05:38:23.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-mon[33554]: pgmap v1173: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:38:23.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:22 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[38106]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T05:38:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-152"}]: dispatch 2024-03-26T05:38:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-152"}]: dispatch 2024-03-26T05:38:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[38106]: pgmap v1173: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:38:23.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[29560]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T05:38:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-152"}]: dispatch 2024-03-26T05:38:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-152"}]: dispatch 2024-03-26T05:38:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[29560]: pgmap v1173: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:38:23.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:22 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:23 smithi164 ceph-mon[33554]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T05:38:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:23 smithi136 ceph-mon[29560]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T05:38:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:23 smithi136 ceph-mon[38106]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T05:38:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:24 smithi164 ceph-mon[33554]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T05:38:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:24 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:24 smithi164 ceph-mon[33554]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:24 smithi164 ceph-mon[33554]: pgmap v1176: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:38:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:25.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:24 smithi164 ceph-mon[33554]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T05:38:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[29560]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T05:38:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[29560]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[29560]: pgmap v1176: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:38:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:25.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[29560]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T05:38:25.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[38106]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T05:38:25.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:25.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:38:25.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[38106]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:25.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[38106]: pgmap v1176: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:38:25.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-75071-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:38:25.328 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:24 smithi136 ceph-mon[38106]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T05:38:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-111","var": "dedup_tier","val": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-111","var": "dedup_tier","val": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:25 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:25 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:25 smithi164 ceph-mon[33554]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T05:38:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-111","var": "dedup_tier","val": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-111","var": "dedup_tier","val": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[29560]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T05:38:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-111","var": "dedup_tier","val": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi136-75071-111","var": "dedup_tier","val": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi136-75071-111", "tierpool": "test-rados-api-smithi136-75071-154"}]: dispatch 2024-03-26T05:38:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:25 smithi136 ceph-mon[38106]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T05:38:27.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:26 smithi136 ceph-mon[29560]: pgmap v1179: 204 pgs: 204 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T05:38:27.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:26 smithi136 ceph-mon[38106]: pgmap v1179: 204 pgs: 204 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T05:38:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:26 smithi164 ceph-mon[33554]: pgmap v1179: 204 pgs: 204 active+clean; 4.4 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T05:38:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:38:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:38:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[29560]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[38106]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:27 smithi164 ceph-mon[33554]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T05:38:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:27 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:27 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:38:28.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7910 ms) 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12168 ms) 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8059 ms) 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13081 ms) 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-26T05:38:28.998 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7703 ms) 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8049 ms) 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: ok, hit_set contains 324:602f83fe:::foo:head 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9056 ms) 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,0 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: first is 1711431425 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,0 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,0 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,0 2024-03-26T05:38:28.999 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,1711431430,1711431431,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,1711431430,1711431431,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,1711431430,1711431431,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,1711431430,1711431431,1711431433,1711431434,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,1711431430,1711431431,1711431433,1711431434,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431425,1711431427,1711431428,1711431430,1711431431,1711431433,1711431434,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: got ls 1711431428,1711431430,1711431431,1711431433,1711431434,1711431436,1711431437,0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: first now 1711431428, trimmed 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21092 ms) 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: foo0 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T05:38:29.000 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14088 ms) 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17437 ms) 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22133 ms) 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5044 ms) 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3018 ms) 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3026 ms) 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3019 ms) 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (229480 ms total) 2024-03-26T05:38:29.001 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: 2024-03-26T05:38:29.002 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-26T05:38:29.002 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (857130 ms total) 2024-03-26T05:38:29.002 INFO:tasks.workunit.client.0.smithi136.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-26T05:38:29.003 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.003 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74738 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74738 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74884 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74884 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75465 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75465 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75136 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75136 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75018 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75018 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74813 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74813 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75076 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75076 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75616 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75616 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75651 2024-03-26T05:38:29.005 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75651 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74831 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74831 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75500 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75500 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75328 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75328 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74709 2024-03-26T05:38:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74709 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74772 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74772 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75375 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75375 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74700 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74700 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74720 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74720 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75754 2024-03-26T05:38:29.007 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75754 2024-03-26T05:38:29.008 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.008 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74905 2024-03-26T05:38:29.008 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74905 2024-03-26T05:38:29.008 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:38:29.008 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75827 2024-03-26T05:38:29.008 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75827 2024-03-26T05:38:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[38106]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T05:38:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[38106]: pgmap v1182: 196 pgs: 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[29560]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T05:38:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:29.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:28 smithi136 ceph-mon[29560]: pgmap v1182: 196 pgs: 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:28 smithi164 ceph-mon[33554]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T05:38:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:28 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2544667150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:28 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]: dispatch 2024-03-26T05:38:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:28 smithi164 ceph-mon[33554]: pgmap v1182: 196 pgs: 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:29 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:29 smithi136 ceph-mon[38106]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T05:38:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:29 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:30.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:29 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:29 smithi136 ceph-mon[29560]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T05:38:30.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:29 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:29 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi136-75071-111"}]': finished 2024-03-26T05:38:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:29 smithi164 ceph-mon[33554]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T05:38:30.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:29 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:38:31.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:31 smithi136 ceph-mon[38106]: pgmap v1184: 196 pgs: 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:31.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:31 smithi136 ceph-mon[29560]: pgmap v1184: 196 pgs: 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:31.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:31 smithi164 ceph-mon[33554]: pgmap v1184: 196 pgs: 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:32.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:38:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:38:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:33 smithi164 ceph-mon[33554]: pgmap v1185: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:33.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:33 smithi136 ceph-mon[38106]: pgmap v1185: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:33.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:33 smithi136 ceph-mon[29560]: pgmap v1185: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:38:33.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:35.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:35 smithi164 ceph-mon[33554]: pgmap v1186: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-26T05:38:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:35 smithi136 ceph-mon[38106]: pgmap v1186: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-26T05:38:35.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:35 smithi136 ceph-mon[29560]: pgmap v1186: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-26T05:38:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:37 smithi164 ceph-mon[33554]: pgmap v1187: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:37 smithi136 ceph-mon[38106]: pgmap v1187: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:37 smithi136 ceph-mon[29560]: pgmap v1187: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:38:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:38:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:38:39.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:39 smithi164 ceph-mon[33554]: pgmap v1188: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:38:39.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:39 smithi136 ceph-mon[38106]: pgmap v1188: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:38:39.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:39 smithi136 ceph-mon[29560]: pgmap v1188: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:38:41.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:41 smithi164 ceph-mon[33554]: pgmap v1189: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-26T05:38:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:41 smithi136 ceph-mon[38106]: pgmap v1189: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-26T05:38:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:41 smithi136 ceph-mon[29560]: pgmap v1189: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-26T05:38:42.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:38:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:38:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:43 smithi164 ceph-mon[33554]: pgmap v1190: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:43.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:43 smithi136 ceph-mon[38106]: pgmap v1190: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:43.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:43 smithi136 ceph-mon[29560]: pgmap v1190: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:43.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:43.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:45 smithi164 ceph-mon[33554]: pgmap v1191: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:45.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:45 smithi136 ceph-mon[38106]: pgmap v1191: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:45.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:45 smithi136 ceph-mon[29560]: pgmap v1191: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:47.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:47 smithi164 ceph-mon[33554]: pgmap v1192: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:47 smithi136 ceph-mon[38106]: pgmap v1192: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:38:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:38:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:38:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:47 smithi136 ceph-mon[29560]: pgmap v1192: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:49.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:49 smithi164 ceph-mon[33554]: pgmap v1193: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:49.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:49 smithi136 ceph-mon[38106]: pgmap v1193: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:49.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:49 smithi136 ceph-mon[29560]: pgmap v1193: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:51.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:51 smithi164 ceph-mon[33554]: pgmap v1194: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:51.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:51 smithi136 ceph-mon[38106]: pgmap v1194: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:51.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:51 smithi136 ceph-mon[29560]: pgmap v1194: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:52.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:38:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:38:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:53 smithi164 ceph-mon[33554]: pgmap v1195: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:53 smithi136 ceph-mon[38106]: pgmap v1195: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:53 smithi136 ceph-mon[29560]: pgmap v1195: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:53.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:38:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:55 smithi164 ceph-mon[33554]: pgmap v1196: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:55.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:55 smithi136 ceph-mon[29560]: pgmap v1196: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:55 smithi136 ceph-mon[38106]: pgmap v1196: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:57 smithi164 ceph-mon[33554]: pgmap v1197: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:57 smithi136 ceph-mon[29560]: pgmap v1197: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:57.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:57 smithi136 ceph-mon[38106]: pgmap v1197: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:38:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:38:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:38:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:38:58.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:38:59.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:38:59 smithi136 ceph-mon[38106]: pgmap v1198: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:38:59 smithi136 ceph-mon[29560]: pgmap v1198: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:38:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:38:59 smithi164 ceph-mon[33554]: pgmap v1198: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:01.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:01 smithi136 ceph-mon[29560]: pgmap v1199: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:01 smithi136 ceph-mon[38106]: pgmap v1199: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:01.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:01 smithi164 ceph-mon[33554]: pgmap v1199: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:02.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:39:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:39:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:03 smithi136 ceph-mon[38106]: pgmap v1200: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:03 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:39:03.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:03 smithi136 ceph-mon[29560]: pgmap v1200: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:03.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:03.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:03 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:39:03.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:03 smithi164 ceph-mon[33554]: pgmap v1200: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:03.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:03.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:03 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:39:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:04 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:39:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:04 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:39:04.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:04 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:39:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:04 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:39:04.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:04 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:39:04.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:04 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:39:04.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:04 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:39:04.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:04 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:39:04.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:04 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:39:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:05 smithi136 ceph-mon[38106]: pgmap v1201: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:05 smithi136 ceph-mon[29560]: pgmap v1201: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:05 smithi164 ceph-mon[33554]: pgmap v1201: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:39:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:39:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:39:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:07 smithi136 ceph-mon[38106]: pgmap v1202: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:08.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:07 smithi136 ceph-mon[29560]: pgmap v1202: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:07 smithi164 ceph-mon[33554]: pgmap v1202: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:09 smithi136 ceph-mon[38106]: pgmap v1203: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:09 smithi136 ceph-mon[29560]: pgmap v1203: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:09 smithi164 ceph-mon[33554]: pgmap v1203: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:12.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:11 smithi136 ceph-mon[29560]: pgmap v1204: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:12.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:11 smithi136 ceph-mon[38106]: pgmap v1204: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:11 smithi164 ceph-mon[33554]: pgmap v1204: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:13.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:39:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:39:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:13 smithi136 ceph-mon[38106]: pgmap v1205: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:13 smithi136 ceph-mon[29560]: pgmap v1205: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:13 smithi164 ceph-mon[33554]: pgmap v1205: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:15 smithi136 ceph-mon[38106]: pgmap v1206: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:15 smithi136 ceph-mon[29560]: pgmap v1206: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:15 smithi164 ceph-mon[33554]: pgmap v1206: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:39:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:39:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:39:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:17 smithi136 ceph-mon[29560]: pgmap v1207: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:17 smithi136 ceph-mon[38106]: pgmap v1207: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:17 smithi164 ceph-mon[33554]: pgmap v1207: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:20.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:19 smithi136 ceph-mon[29560]: pgmap v1208: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:39:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:19 smithi136 ceph-mon[38106]: pgmap v1208: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:39:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:19 smithi164 ceph-mon[33554]: pgmap v1208: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:39:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:21 smithi136 ceph-mon[38106]: pgmap v1209: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:21 smithi136 ceph-mon[29560]: pgmap v1209: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:21 smithi164 ceph-mon[33554]: pgmap v1209: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:23.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:39:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:39:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:23 smithi136 ceph-mon[38106]: pgmap v1210: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:23 smithi136 ceph-mon[29560]: pgmap v1210: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:23 smithi164 ceph-mon[33554]: pgmap v1210: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:25 smithi136 ceph-mon[29560]: pgmap v1211: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:39:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:25 smithi136 ceph-mon[38106]: pgmap v1211: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:39:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:25 smithi164 ceph-mon[33554]: pgmap v1211: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:39:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:39:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:39:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:39:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:27 smithi136 ceph-mon[38106]: pgmap v1212: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:27 smithi136 ceph-mon[29560]: pgmap v1212: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:27 smithi164 ceph-mon[33554]: pgmap v1212: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:29 smithi136 ceph-mon[38106]: pgmap v1213: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:29 smithi136 ceph-mon[29560]: pgmap v1213: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:29 smithi164 ceph-mon[33554]: pgmap v1213: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:31 smithi136 ceph-mon[38106]: pgmap v1214: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:31 smithi136 ceph-mon[29560]: pgmap v1214: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:31 smithi164 ceph-mon[33554]: pgmap v1214: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:33.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:39:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:39:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:33 smithi136 ceph-mon[38106]: pgmap v1215: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:33 smithi136 ceph-mon[29560]: pgmap v1215: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:33 smithi164 ceph-mon[33554]: pgmap v1215: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:35 smithi136 ceph-mon[38106]: pgmap v1216: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:35 smithi136 ceph-mon[29560]: pgmap v1216: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:35 smithi164 ceph-mon[33554]: pgmap v1216: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:39:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:39:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:39:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:37 smithi136 ceph-mon[38106]: pgmap v1217: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:37 smithi136 ceph-mon[29560]: pgmap v1217: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:37 smithi164 ceph-mon[33554]: pgmap v1217: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:39 smithi136 ceph-mon[38106]: pgmap v1218: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:39 smithi136 ceph-mon[29560]: pgmap v1218: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:39 smithi164 ceph-mon[33554]: pgmap v1218: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:41 smithi136 ceph-mon[38106]: pgmap v1219: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:41 smithi136 ceph-mon[29560]: pgmap v1219: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:41 smithi164 ceph-mon[33554]: pgmap v1219: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:42 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:43.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:39:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:39:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:42 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:42 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:43 smithi164 ceph-mon[33554]: pgmap v1220: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:43 smithi136 ceph-mon[38106]: pgmap v1220: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:43 smithi136 ceph-mon[29560]: pgmap v1220: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:45 smithi164 ceph-mon[33554]: pgmap v1221: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:45 smithi136 ceph-mon[38106]: pgmap v1221: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:46.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:45 smithi136 ceph-mon[29560]: pgmap v1221: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:39:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:39:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:39:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:47 smithi164 ceph-mon[33554]: pgmap v1222: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:47 smithi136 ceph-mon[38106]: pgmap v1222: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:48.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:47 smithi136 ceph-mon[29560]: pgmap v1222: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:49 smithi164 ceph-mon[33554]: pgmap v1223: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:49 smithi136 ceph-mon[38106]: pgmap v1223: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:49 smithi136 ceph-mon[29560]: pgmap v1223: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:51 smithi164 ceph-mon[33554]: pgmap v1224: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:51 smithi136 ceph-mon[38106]: pgmap v1224: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:51 smithi136 ceph-mon[29560]: pgmap v1224: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:53.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:39:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:39:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:53 smithi164 ceph-mon[33554]: pgmap v1225: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:53 smithi136 ceph-mon[38106]: pgmap v1225: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:53 smithi136 ceph-mon[29560]: pgmap v1225: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:39:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:55 smithi164 ceph-mon[33554]: pgmap v1226: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:55 smithi136 ceph-mon[29560]: pgmap v1226: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:55 smithi136 ceph-mon[38106]: pgmap v1226: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:39:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:39:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:39:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:39:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:57 smithi164 ceph-mon[33554]: pgmap v1227: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:57 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:57 smithi136 ceph-mon[29560]: pgmap v1227: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:57 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:39:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:57 smithi136 ceph-mon[38106]: pgmap v1227: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:39:58.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:57 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:39:59 smithi164 ceph-mon[33554]: pgmap v1228: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:00.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:39:59 smithi136 ceph-mon[38106]: pgmap v1228: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:00.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:39:59 smithi136 ceph-mon[29560]: pgmap v1228: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:01.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:00 smithi164 ceph-mon[33554]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:40:01.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:00 smithi136 ceph-mon[29560]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:40:01.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:00 smithi136 ceph-mon[38106]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:40:02.222 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:01 smithi164 ceph-mon[33554]: pgmap v1229: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:01 smithi136 ceph-mon[38106]: pgmap v1229: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:02.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:01 smithi136 ceph-mon[29560]: pgmap v1229: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:03.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:40:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:40:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:03 smithi164 ceph-mon[33554]: pgmap v1230: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:03 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:40:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:03 smithi136 ceph-mon[38106]: pgmap v1230: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:03 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:40:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:03 smithi136 ceph-mon[29560]: pgmap v1230: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:04.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:04.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:03 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:40:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:05 smithi136 ceph-mon[38106]: pgmap v1231: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:05 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:40:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:05 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:40:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:05 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:40:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:05 smithi136 ceph-mon[29560]: pgmap v1231: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:05.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:05 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:40:05.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:05 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:40:05.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:05 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:40:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:05 smithi164 ceph-mon[33554]: pgmap v1231: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:05 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:40:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:05 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:40:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:05 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:40:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:07 smithi164 ceph-mon[33554]: pgmap v1232: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:07 smithi136 ceph-mon[38106]: pgmap v1232: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:07 smithi136 ceph-mon[29560]: pgmap v1232: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:07.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:40:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:40:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:40:09.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:09 smithi164 ceph-mon[33554]: pgmap v1233: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:09.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:09 smithi136 ceph-mon[38106]: pgmap v1233: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:09.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:09 smithi136 ceph-mon[29560]: pgmap v1233: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:11.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:11 smithi164 ceph-mon[33554]: pgmap v1234: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:11 smithi136 ceph-mon[38106]: pgmap v1234: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:11.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:11 smithi136 ceph-mon[29560]: pgmap v1234: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:13.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:40:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:40:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:13 smithi136 ceph-mon[38106]: pgmap v1235: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:13 smithi136 ceph-mon[29560]: pgmap v1235: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:13 smithi164 ceph-mon[33554]: pgmap v1235: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:15 smithi136 ceph-mon[38106]: pgmap v1236: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:15.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:15 smithi136 ceph-mon[29560]: pgmap v1236: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:15 smithi164 ceph-mon[33554]: pgmap v1236: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:17 smithi136 ceph-mon[38106]: pgmap v1237: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:17.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:17 smithi136 ceph-mon[29560]: pgmap v1237: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:40:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:40:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:40:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:17 smithi164 ceph-mon[33554]: pgmap v1237: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:19.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:19 smithi136 ceph-mon[38106]: pgmap v1238: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:19.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:19 smithi136 ceph-mon[29560]: pgmap v1238: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:19 smithi164 ceph-mon[33554]: pgmap v1238: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:21 smithi136 ceph-mon[38106]: pgmap v1239: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:21.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:21 smithi136 ceph-mon[29560]: pgmap v1239: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:21 smithi164 ceph-mon[33554]: pgmap v1239: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:23.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:40:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:40:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:23 smithi136 ceph-mon[29560]: pgmap v1240: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:23.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:23 smithi136 ceph-mon[38106]: pgmap v1240: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:23.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:23.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:23 smithi164 ceph-mon[33554]: pgmap v1240: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:23.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:25.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:25 smithi136 ceph-mon[38106]: pgmap v1241: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:25.579 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:25 smithi136 ceph-mon[29560]: pgmap v1241: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:25 smithi164 ceph-mon[33554]: pgmap v1241: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:27 smithi136 ceph-mon[38106]: pgmap v1242: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:27 smithi136 ceph-mon[29560]: pgmap v1242: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:40:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:40:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:40:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:27 smithi164 ceph-mon[33554]: pgmap v1242: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:28.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:29 smithi136 ceph-mon[38106]: pgmap v1243: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:29.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:29 smithi136 ceph-mon[29560]: pgmap v1243: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:29 smithi164 ceph-mon[33554]: pgmap v1243: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:31 smithi136 ceph-mon[38106]: pgmap v1244: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:31.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:31 smithi136 ceph-mon[29560]: pgmap v1244: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:31 smithi164 ceph-mon[33554]: pgmap v1244: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:33.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:40:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:40:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:33 smithi136 ceph-mon[38106]: pgmap v1245: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:33.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:33 smithi136 ceph-mon[29560]: pgmap v1245: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:33.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:33 smithi164 ceph-mon[33554]: pgmap v1245: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:35 smithi136 ceph-mon[38106]: pgmap v1246: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:35.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:35 smithi136 ceph-mon[29560]: pgmap v1246: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:35 smithi164 ceph-mon[33554]: pgmap v1246: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:37 smithi136 ceph-mon[29560]: pgmap v1247: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:37 smithi136 ceph-mon[38106]: pgmap v1247: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:40:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:40:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:40:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:37 smithi164 ceph-mon[33554]: pgmap v1247: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:39.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:39 smithi136 ceph-mon[29560]: pgmap v1248: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:39.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:39 smithi136 ceph-mon[38106]: pgmap v1248: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:39 smithi164 ceph-mon[33554]: pgmap v1248: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:41 smithi136 ceph-mon[38106]: pgmap v1249: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:41 smithi136 ceph-mon[29560]: pgmap v1249: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:41 smithi164 ceph-mon[33554]: pgmap v1249: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:43.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:40:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:40:43.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:43 smithi136 ceph-mon[29560]: pgmap v1250: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:43.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:43.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:43 smithi136 ceph-mon[38106]: pgmap v1250: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:43.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:43.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:43 smithi164 ceph-mon[33554]: pgmap v1250: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:45.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:45 smithi136 ceph-mon[38106]: pgmap v1251: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:45.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:45 smithi136 ceph-mon[29560]: pgmap v1251: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:45 smithi164 ceph-mon[33554]: pgmap v1251: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:40:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:47 smithi136 ceph-mon[38106]: pgmap v1252: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:40:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:40:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:40:47.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:47 smithi136 ceph-mon[29560]: pgmap v1252: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:47 smithi164 ceph-mon[33554]: pgmap v1252: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:49.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:49 smithi136 ceph-mon[38106]: pgmap v1253: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:49.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:49 smithi136 ceph-mon[29560]: pgmap v1253: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:49 smithi164 ceph-mon[33554]: pgmap v1253: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:51.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:51 smithi136 ceph-mon[38106]: pgmap v1254: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:51.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:51 smithi136 ceph-mon[29560]: pgmap v1254: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:51.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:51 smithi164 ceph-mon[33554]: pgmap v1254: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:53.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:40:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:40:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:53 smithi136 ceph-mon[38106]: pgmap v1255: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:53.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:53.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:53 smithi136 ceph-mon[29560]: pgmap v1255: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:53.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:53 smithi164 ceph-mon[33554]: pgmap v1255: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:40:55.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:55 smithi136 ceph-mon[38106]: pgmap v1256: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:55.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:55 smithi136 ceph-mon[29560]: pgmap v1256: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:55 smithi164 ceph-mon[33554]: pgmap v1256: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:57.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:57 smithi136 ceph-mon[38106]: pgmap v1257: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:57 smithi136 ceph-mon[29560]: pgmap v1257: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:57.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:40:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:40:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:40:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:57 smithi164 ceph-mon[33554]: pgmap v1257: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:40:58.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:58.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:40:59.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:40:59 smithi136 ceph-mon[38106]: pgmap v1258: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:59.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:40:59 smithi136 ceph-mon[29560]: pgmap v1258: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:40:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:40:59 smithi164 ceph-mon[33554]: pgmap v1258: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:01.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:01 smithi136 ceph-mon[38106]: pgmap v1259: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:01.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:01 smithi136 ceph-mon[29560]: pgmap v1259: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:01.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:01 smithi164 ceph-mon[33554]: pgmap v1259: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:03.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:41:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:41:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:03 smithi136 ceph-mon[38106]: pgmap v1260: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:03.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:03.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:03 smithi136 ceph-mon[29560]: pgmap v1260: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:03.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:03.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:03 smithi164 ceph-mon[33554]: pgmap v1260: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:03.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:05.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[38106]: pgmap v1261: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:05.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:41:05.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:41:05.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:41:05.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:41:05.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[29560]: pgmap v1261: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:05.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:41:05.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:41:05.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:41:05.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:05 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:41:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:05 smithi164 ceph-mon[33554]: pgmap v1261: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:05 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:41:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:05 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:41:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:05 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:41:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:05 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:41:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:07 smithi136 ceph-mon[29560]: pgmap v1262: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:07 smithi136 ceph-mon[38106]: pgmap v1262: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:41:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:41:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:41:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:07 smithi164 ceph-mon[33554]: pgmap v1262: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:09.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:09 smithi136 ceph-mon[29560]: pgmap v1263: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:09.607 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:09 smithi136 ceph-mon[38106]: pgmap v1263: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:09.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:09 smithi164 ceph-mon[33554]: pgmap v1263: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:11.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:11 smithi136 ceph-mon[29560]: pgmap v1264: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:11 smithi136 ceph-mon[38106]: pgmap v1264: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:11 smithi164 ceph-mon[33554]: pgmap v1264: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:13.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:41:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:41:13.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:13 smithi164 ceph-mon[33554]: pgmap v1265: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:13.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:13.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:13 smithi136 ceph-mon[29560]: pgmap v1265: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:13 smithi136 ceph-mon[38106]: pgmap v1265: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:15 smithi164 ceph-mon[33554]: pgmap v1266: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:15 smithi136 ceph-mon[29560]: pgmap v1266: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:15 smithi136 ceph-mon[38106]: pgmap v1266: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:17 smithi136 ceph-mon[38106]: pgmap v1267: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:17.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:17 smithi136 ceph-mon[29560]: pgmap v1267: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:17.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:41:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:41:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:41:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:17 smithi164 ceph-mon[33554]: pgmap v1267: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:19 smithi164 ceph-mon[33554]: pgmap v1268: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:41:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:19 smithi136 ceph-mon[38106]: pgmap v1268: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:41:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:19 smithi136 ceph-mon[29560]: pgmap v1268: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:41:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:21 smithi164 ceph-mon[33554]: pgmap v1269: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:21 smithi136 ceph-mon[38106]: pgmap v1269: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:21 smithi136 ceph-mon[29560]: pgmap v1269: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:23.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:41:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:41:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:23 smithi164 ceph-mon[33554]: pgmap v1270: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:23 smithi136 ceph-mon[38106]: pgmap v1270: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:23 smithi136 ceph-mon[29560]: pgmap v1270: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:25 smithi164 ceph-mon[33554]: pgmap v1271: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:25 smithi136 ceph-mon[38106]: pgmap v1271: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:25 smithi136 ceph-mon[29560]: pgmap v1271: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:27 smithi136 ceph-mon[29560]: pgmap v1272: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:27 smithi136 ceph-mon[38106]: pgmap v1272: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:27.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:41:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:41:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:41:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:27 smithi164 ceph-mon[33554]: pgmap v1272: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:28.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:29 smithi164 ceph-mon[33554]: pgmap v1273: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:29 smithi136 ceph-mon[38106]: pgmap v1273: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:29 smithi136 ceph-mon[29560]: pgmap v1273: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:31 smithi164 ceph-mon[33554]: pgmap v1274: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:31 smithi136 ceph-mon[38106]: pgmap v1274: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:31.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:31 smithi136 ceph-mon[29560]: pgmap v1274: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:33.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:41:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:41:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:33 smithi164 ceph-mon[33554]: pgmap v1275: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:33 smithi136 ceph-mon[38106]: pgmap v1275: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:33 smithi136 ceph-mon[29560]: pgmap v1275: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:35 smithi164 ceph-mon[33554]: pgmap v1276: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:35 smithi136 ceph-mon[38106]: pgmap v1276: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:35 smithi136 ceph-mon[29560]: pgmap v1276: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:37 smithi136 ceph-mon[38106]: pgmap v1277: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:37 smithi136 ceph-mon[29560]: pgmap v1277: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:41:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:41:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:41:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:37 smithi164 ceph-mon[33554]: pgmap v1277: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:39 smithi164 ceph-mon[33554]: pgmap v1278: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:39 smithi136 ceph-mon[38106]: pgmap v1278: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:39 smithi136 ceph-mon[29560]: pgmap v1278: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:41.626 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:41 smithi136 ceph-mon[38106]: pgmap v1279: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:41.627 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:41 smithi136 ceph-mon[29560]: pgmap v1279: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:41 smithi164 ceph-mon[33554]: pgmap v1279: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:41:43.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:41:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:41:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:43 smithi164 ceph-mon[33554]: pgmap v1280: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:43.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:43 smithi136 ceph-mon[29560]: pgmap v1280: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:43 smithi136 ceph-mon[38106]: pgmap v1280: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:43.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:45 smithi164 ceph-mon[33554]: pgmap v1281: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:45 smithi136 ceph-mon[29560]: pgmap v1281: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:45 smithi136 ceph-mon[38106]: pgmap v1281: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:47 smithi136 ceph-mon[38106]: pgmap v1282: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:47 smithi136 ceph-mon[29560]: pgmap v1282: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:41:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:41:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:41:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:47 smithi164 ceph-mon[33554]: pgmap v1282: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:49 smithi164 ceph-mon[33554]: pgmap v1283: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:49 smithi136 ceph-mon[38106]: pgmap v1283: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:49.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:49 smithi136 ceph-mon[29560]: pgmap v1283: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:51.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:51 smithi164 ceph-mon[33554]: pgmap v1284: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:51 smithi136 ceph-mon[38106]: pgmap v1284: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:51.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:51 smithi136 ceph-mon[29560]: pgmap v1284: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:53.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:41:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:41:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:53 smithi164 ceph-mon[33554]: pgmap v1285: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:53 smithi136 ceph-mon[38106]: pgmap v1285: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:53 smithi136 ceph-mon[29560]: pgmap v1285: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:41:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:41:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:55 smithi164 ceph-mon[33554]: pgmap v1286: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:41:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:55 smithi136 ceph-mon[38106]: pgmap v1286: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:41:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:55 smithi136 ceph-mon[29560]: pgmap v1286: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:41:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:57 smithi136 ceph-mon[29560]: pgmap v1287: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:41:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:41:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:41:57.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:57 smithi136 ceph-mon[38106]: pgmap v1287: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:57 smithi164 ceph-mon[33554]: pgmap v1287: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:41:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:41:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:41:59 smithi164 ceph-mon[33554]: pgmap v1288: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:41:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:41:59 smithi136 ceph-mon[29560]: pgmap v1288: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:41:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:41:59 smithi136 ceph-mon[38106]: pgmap v1288: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:01.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:01 smithi164 ceph-mon[33554]: pgmap v1289: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:01 smithi136 ceph-mon[38106]: pgmap v1289: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:01 smithi136 ceph-mon[29560]: pgmap v1289: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:03.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:42:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:42:03.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:03 smithi164 ceph-mon[33554]: pgmap v1290: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:03.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:03 smithi136 ceph-mon[38106]: pgmap v1290: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:03 smithi136 ceph-mon[29560]: pgmap v1290: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:05 smithi164 ceph-mon[33554]: pgmap v1291: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:05 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:42:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:05 smithi136 ceph-mon[38106]: pgmap v1291: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:05 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:42:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:05 smithi136 ceph-mon[29560]: pgmap v1291: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:05 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:42:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:06 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:42:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:06 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:42:06.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:06 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:42:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:06 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:42:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:06 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:42:06.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:06 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:42:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:06 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:42:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:06 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:42:06.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:06 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:42:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:42:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:42:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:42:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:07 smithi164 ceph-mon[33554]: pgmap v1292: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:08.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:07 smithi136 ceph-mon[29560]: pgmap v1292: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:08.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:07 smithi136 ceph-mon[38106]: pgmap v1292: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:09 smithi164 ceph-mon[33554]: pgmap v1293: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:09 smithi136 ceph-mon[29560]: pgmap v1293: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:09 smithi136 ceph-mon[38106]: pgmap v1293: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:11 smithi164 ceph-mon[33554]: pgmap v1294: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:11 smithi136 ceph-mon[29560]: pgmap v1294: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:11 smithi136 ceph-mon[38106]: pgmap v1294: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:13.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:42:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:42:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:13 smithi164 ceph-mon[33554]: pgmap v1295: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:13 smithi136 ceph-mon[38106]: pgmap v1295: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:13 smithi136 ceph-mon[29560]: pgmap v1295: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:15 smithi164 ceph-mon[33554]: pgmap v1296: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:15 smithi136 ceph-mon[38106]: pgmap v1296: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:16.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:15 smithi136 ceph-mon[29560]: pgmap v1296: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:42:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:42:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:42:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:17 smithi164 ceph-mon[33554]: pgmap v1297: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:17 smithi136 ceph-mon[38106]: pgmap v1297: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:18.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:17 smithi136 ceph-mon[29560]: pgmap v1297: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:19 smithi164 ceph-mon[33554]: pgmap v1298: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:19 smithi136 ceph-mon[38106]: pgmap v1298: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:20.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:19 smithi136 ceph-mon[29560]: pgmap v1298: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:21 smithi164 ceph-mon[33554]: pgmap v1299: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:21 smithi136 ceph-mon[38106]: pgmap v1299: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:21 smithi136 ceph-mon[29560]: pgmap v1299: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:23.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:42:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:42:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:23 smithi164 ceph-mon[33554]: pgmap v1300: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:23 smithi136 ceph-mon[38106]: pgmap v1300: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:23 smithi136 ceph-mon[29560]: pgmap v1300: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:25 smithi164 ceph-mon[33554]: pgmap v1301: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:25 smithi136 ceph-mon[38106]: pgmap v1301: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:25 smithi136 ceph-mon[29560]: pgmap v1301: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:42:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:42:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:42:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:27 smithi164 ceph-mon[33554]: pgmap v1302: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:27 smithi136 ceph-mon[38106]: pgmap v1302: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:27 smithi136 ceph-mon[29560]: pgmap v1302: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:29 smithi164 ceph-mon[33554]: pgmap v1303: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:29 smithi136 ceph-mon[38106]: pgmap v1303: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:30.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:29 smithi136 ceph-mon[29560]: pgmap v1303: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:31 smithi164 ceph-mon[33554]: pgmap v1304: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:31 smithi136 ceph-mon[29560]: pgmap v1304: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:31 smithi136 ceph-mon[38106]: pgmap v1304: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:33.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:42:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:42:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:33 smithi164 ceph-mon[33554]: pgmap v1305: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:33 smithi136 ceph-mon[29560]: pgmap v1305: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:33 smithi136 ceph-mon[38106]: pgmap v1305: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:35 smithi164 ceph-mon[33554]: pgmap v1306: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:35 smithi136 ceph-mon[29560]: pgmap v1306: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:35 smithi136 ceph-mon[38106]: pgmap v1306: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:42:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:42:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:42:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:37 smithi164 ceph-mon[33554]: pgmap v1307: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:37 smithi136 ceph-mon[38106]: pgmap v1307: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:37 smithi136 ceph-mon[29560]: pgmap v1307: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:39 smithi164 ceph-mon[33554]: pgmap v1308: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:39 smithi136 ceph-mon[38106]: pgmap v1308: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:39 smithi136 ceph-mon[29560]: pgmap v1308: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:41 smithi164 ceph-mon[33554]: pgmap v1309: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:41 smithi136 ceph-mon[38106]: pgmap v1309: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:41 smithi136 ceph-mon[29560]: pgmap v1309: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:43.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:42:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:42:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:43 smithi164 ceph-mon[33554]: pgmap v1310: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:43 smithi136 ceph-mon[38106]: pgmap v1310: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:43 smithi136 ceph-mon[29560]: pgmap v1310: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:44.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:45 smithi164 ceph-mon[33554]: pgmap v1311: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:45 smithi136 ceph-mon[38106]: pgmap v1311: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:46.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:45 smithi136 ceph-mon[29560]: pgmap v1311: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:42:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:42:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:42:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:47 smithi164 ceph-mon[33554]: pgmap v1312: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:48.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:47 smithi136 ceph-mon[38106]: pgmap v1312: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:47 smithi136 ceph-mon[29560]: pgmap v1312: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:49 smithi164 ceph-mon[33554]: pgmap v1313: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:49 smithi136 ceph-mon[29560]: pgmap v1313: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:49 smithi136 ceph-mon[38106]: pgmap v1313: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:51 smithi164 ceph-mon[33554]: pgmap v1314: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:51 smithi136 ceph-mon[29560]: pgmap v1314: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:51 smithi136 ceph-mon[38106]: pgmap v1314: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:42:53.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:42:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:42:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:53 smithi164 ceph-mon[33554]: pgmap v1315: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:53 smithi136 ceph-mon[38106]: pgmap v1315: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:53 smithi136 ceph-mon[29560]: pgmap v1315: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:42:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:55 smithi164 ceph-mon[33554]: pgmap v1316: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:55 smithi136 ceph-mon[29560]: pgmap v1316: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:55 smithi136 ceph-mon[38106]: pgmap v1316: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:42:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:42:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:42:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:42:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:57 smithi164 ceph-mon[33554]: pgmap v1317: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:57 smithi136 ceph-mon[38106]: pgmap v1317: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:57 smithi136 ceph-mon[29560]: pgmap v1317: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:42:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:42:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:42:59 smithi164 ceph-mon[33554]: pgmap v1318: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:43:00.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:42:59 smithi136 ceph-mon[38106]: pgmap v1318: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:43:00.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:42:59 smithi136 ceph-mon[29560]: pgmap v1318: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:43:02.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:01 smithi164 ceph-mon[33554]: pgmap v1319: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:01 smithi136 ceph-mon[38106]: pgmap v1319: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:02.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:01 smithi136 ceph-mon[29560]: pgmap v1319: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:03.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:43:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:43:03.719 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:43:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=cleanup t=2024-03-26T05:43:03.352749394Z level=info msg="Completed cleanup jobs" duration=2.388643ms 2024-03-26T05:43:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:03 smithi164 ceph-mon[33554]: pgmap v1320: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:03 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:03 smithi136 ceph-mon[38106]: pgmap v1320: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:03 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:03 smithi136 ceph-mon[29560]: pgmap v1320: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:03 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:05 smithi164 ceph-mon[33554]: pgmap v1321: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:43:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:05 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:43:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:05 smithi136 ceph-mon[38106]: pgmap v1321: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:43:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:05 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:43:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:05 smithi136 ceph-mon[29560]: pgmap v1321: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:43:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:05 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:43:07.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:06 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:43:07.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:06 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:43:07.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:06 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:43:07.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:06 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:43:07.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:06 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:43:07.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:06 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:43:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:06 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:43:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:06 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:43:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:06 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:43:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:43:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:43:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:43:08.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:07 smithi136 ceph-mon[29560]: pgmap v1322: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:08.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:07 smithi136 ceph-mon[38106]: pgmap v1322: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:08.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:07 smithi164 ceph-mon[33554]: pgmap v1322: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:09.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:08 smithi136 ceph-mon[38106]: pgmap v1323: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:09.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:08 smithi136 ceph-mon[29560]: pgmap v1323: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:09.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:08 smithi164 ceph-mon[33554]: pgmap v1323: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:11 smithi136 ceph-mon[38106]: pgmap v1324: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:11.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:11 smithi136 ceph-mon[29560]: pgmap v1324: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:11 smithi164 ceph-mon[33554]: pgmap v1324: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:13.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:43:12 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:43:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:13 smithi136 ceph-mon[38106]: pgmap v1325: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:13 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:13.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:13.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:13 smithi136 ceph-mon[29560]: pgmap v1325: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:13 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:13.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:13 smithi164 ceph-mon[33554]: pgmap v1325: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:13 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:15.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:15 smithi136 ceph-mon[38106]: pgmap v1326: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:15.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:15 smithi136 ceph-mon[29560]: pgmap v1326: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:15 smithi164 ceph-mon[33554]: pgmap v1326: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:17.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:17 smithi136 ceph-mon[29560]: pgmap v1327: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:17 smithi136 ceph-mon[38106]: pgmap v1327: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:43:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:43:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:43:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:17 smithi164 ceph-mon[33554]: pgmap v1327: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:19.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:19 smithi136 ceph-mon[29560]: pgmap v1328: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:19.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:19 smithi136 ceph-mon[38106]: pgmap v1328: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:19 smithi164 ceph-mon[33554]: pgmap v1328: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:21.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:21 smithi136 ceph-mon[29560]: pgmap v1329: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:21.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:21 smithi136 ceph-mon[38106]: pgmap v1329: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:21 smithi164 ceph-mon[33554]: pgmap v1329: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:23.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:43:22 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:43:23.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:23 smithi136 ceph-mon[38106]: pgmap v1330: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:23.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:23 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:23.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:23 smithi136 ceph-mon[29560]: pgmap v1330: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:23.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:23 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:23 smithi164 ceph-mon[33554]: pgmap v1330: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:23 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:25.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:25 smithi136 ceph-mon[29560]: pgmap v1331: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:25 smithi136 ceph-mon[38106]: pgmap v1331: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:25 smithi164 ceph-mon[33554]: pgmap v1331: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:27 smithi136 ceph-mon[29560]: pgmap v1332: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:27 smithi136 ceph-mon[38106]: pgmap v1332: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:27.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:43:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:43:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:43:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:27 smithi164 ceph-mon[33554]: pgmap v1332: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:28.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:28.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:29.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:29 smithi136 ceph-mon[38106]: pgmap v1333: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:29.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:29 smithi136 ceph-mon[29560]: pgmap v1333: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:29 smithi164 ceph-mon[33554]: pgmap v1333: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:31 smithi136 ceph-mon[38106]: pgmap v1334: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:31.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:31 smithi136 ceph-mon[29560]: pgmap v1334: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:31 smithi164 ceph-mon[33554]: pgmap v1334: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:33.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:43:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:43:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:33 smithi136 ceph-mon[38106]: pgmap v1335: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:33.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:33 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:33.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:33 smithi136 ceph-mon[29560]: pgmap v1335: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:33.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:33 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:33 smithi164 ceph-mon[33554]: pgmap v1335: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:33 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:35.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:35 smithi136 ceph-mon[38106]: pgmap v1336: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:35.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:35 smithi136 ceph-mon[29560]: pgmap v1336: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:35 smithi164 ceph-mon[33554]: pgmap v1336: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:43:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:37 smithi136 ceph-mon[38106]: pgmap v1337: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:37 smithi136 ceph-mon[29560]: pgmap v1337: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:43:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:43:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:43:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:37 smithi164 ceph-mon[33554]: pgmap v1337: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:39.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:39 smithi136 ceph-mon[38106]: pgmap v1338: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:39.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:39 smithi136 ceph-mon[29560]: pgmap v1338: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:39 smithi164 ceph-mon[33554]: pgmap v1338: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:41 smithi164 ceph-mon[33554]: pgmap v1339: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:41 smithi136 ceph-mon[38106]: pgmap v1339: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:41 smithi136 ceph-mon[29560]: pgmap v1339: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:43.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:43:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:43:43.627 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:43 smithi136 ceph-mon[38106]: pgmap v1340: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:43.628 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:43 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:43.628 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:43.628 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:43 smithi136 ceph-mon[29560]: pgmap v1340: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:43.628 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:43 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:43.628 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:43 smithi164 ceph-mon[33554]: pgmap v1340: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:43 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:45 smithi164 ceph-mon[33554]: pgmap v1341: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:45 smithi136 ceph-mon[38106]: pgmap v1341: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:45 smithi136 ceph-mon[29560]: pgmap v1341: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:47 smithi136 ceph-mon[29560]: pgmap v1342: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:47 smithi136 ceph-mon[38106]: pgmap v1342: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:43:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:43:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:43:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:47 smithi164 ceph-mon[33554]: pgmap v1342: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:49 smithi164 ceph-mon[33554]: pgmap v1343: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:49 smithi136 ceph-mon[38106]: pgmap v1343: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:49.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:49 smithi136 ceph-mon[29560]: pgmap v1343: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:51.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:51 smithi164 ceph-mon[33554]: pgmap v1344: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:51 smithi136 ceph-mon[38106]: pgmap v1344: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:51.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:51 smithi136 ceph-mon[29560]: pgmap v1344: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:53.352 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:43:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:43:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:53 smithi164 ceph-mon[33554]: pgmap v1345: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:53 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:53 smithi136 ceph-mon[38106]: pgmap v1345: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:53 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:53 smithi136 ceph-mon[29560]: pgmap v1345: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:53 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:43:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:55 smithi164 ceph-mon[33554]: pgmap v1346: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:55 smithi136 ceph-mon[38106]: pgmap v1346: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:55 smithi136 ceph-mon[29560]: pgmap v1346: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:57 smithi136 ceph-mon[29560]: pgmap v1347: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:57.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:57 smithi136 ceph-mon[38106]: pgmap v1347: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:43:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:43:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:43:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:57 smithi164 ceph-mon[33554]: pgmap v1347: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:43:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:43:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:43:59 smithi164 ceph-mon[33554]: pgmap v1348: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:43:59 smithi136 ceph-mon[29560]: pgmap v1348: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:43:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:43:59 smithi136 ceph-mon[38106]: pgmap v1348: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:01.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:01 smithi164 ceph-mon[33554]: pgmap v1349: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:01 smithi136 ceph-mon[38106]: pgmap v1349: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:01 smithi136 ceph-mon[29560]: pgmap v1349: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:03.373 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:44:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:44:03.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:03 smithi164 ceph-mon[33554]: pgmap v1350: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:03 smithi136 ceph-mon[38106]: pgmap v1350: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:03 smithi136 ceph-mon[29560]: pgmap v1350: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:04.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:05.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:05 smithi164 ceph-mon[33554]: pgmap v1351: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:05 smithi136 ceph-mon[38106]: pgmap v1351: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:05 smithi136 ceph-mon[29560]: pgmap v1351: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:44:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:44:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[38106]: pgmap v1352: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:44:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:44:07.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:44:07.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:44:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[29560]: pgmap v1352: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:44:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:44:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:44:07.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:07 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:44:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:07 smithi164 ceph-mon[33554]: pgmap v1352: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:07 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:44:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:07 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:44:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:07 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:44:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:07 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:44:09.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:09 smithi164 ceph-mon[33554]: pgmap v1353: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:09 smithi136 ceph-mon[38106]: pgmap v1353: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:09 smithi136 ceph-mon[29560]: pgmap v1353: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:11 smithi164 ceph-mon[33554]: pgmap v1354: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:11 smithi136 ceph-mon[38106]: pgmap v1354: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:11 smithi136 ceph-mon[29560]: pgmap v1354: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:13.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:44:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:44:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:13 smithi164 ceph-mon[33554]: pgmap v1355: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:13 smithi136 ceph-mon[38106]: pgmap v1355: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:13 smithi136 ceph-mon[29560]: pgmap v1355: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:14.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:14.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:14.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:15 smithi164 ceph-mon[33554]: pgmap v1356: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:15 smithi136 ceph-mon[38106]: pgmap v1356: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:15 smithi136 ceph-mon[29560]: pgmap v1356: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:17 smithi136 ceph-mon[38106]: pgmap v1357: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:17.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:17 smithi136 ceph-mon[29560]: pgmap v1357: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:17.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:44:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:44:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:44:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:17 smithi164 ceph-mon[33554]: pgmap v1357: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:19 smithi164 ceph-mon[33554]: pgmap v1358: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:19 smithi136 ceph-mon[38106]: pgmap v1358: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:19 smithi136 ceph-mon[29560]: pgmap v1358: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:21 smithi164 ceph-mon[33554]: pgmap v1359: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:21 smithi136 ceph-mon[38106]: pgmap v1359: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:21 smithi136 ceph-mon[29560]: pgmap v1359: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:23.415 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:44:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:44:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:23 smithi164 ceph-mon[33554]: pgmap v1360: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:23 smithi136 ceph-mon[38106]: pgmap v1360: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:23 smithi136 ceph-mon[29560]: pgmap v1360: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:24.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:25 smithi164 ceph-mon[33554]: pgmap v1361: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:25 smithi136 ceph-mon[38106]: pgmap v1361: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:25 smithi136 ceph-mon[29560]: pgmap v1361: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:27 smithi136 ceph-mon[38106]: pgmap v1362: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:27 smithi136 ceph-mon[29560]: pgmap v1362: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:27.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:44:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:44:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:44:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:27 smithi164 ceph-mon[33554]: pgmap v1362: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:28.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:29 smithi164 ceph-mon[33554]: pgmap v1363: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:29 smithi136 ceph-mon[38106]: pgmap v1363: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:29 smithi136 ceph-mon[29560]: pgmap v1363: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:31 smithi164 ceph-mon[33554]: pgmap v1364: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:31 smithi136 ceph-mon[38106]: pgmap v1364: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:31.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:31 smithi136 ceph-mon[29560]: pgmap v1364: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:33.443 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:44:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:44:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:33 smithi164 ceph-mon[33554]: pgmap v1365: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:33 smithi136 ceph-mon[29560]: pgmap v1365: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:33 smithi136 ceph-mon[38106]: pgmap v1365: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:34.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:35 smithi164 ceph-mon[33554]: pgmap v1366: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:35 smithi136 ceph-mon[38106]: pgmap v1366: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:35 smithi136 ceph-mon[29560]: pgmap v1366: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:37 smithi136 ceph-mon[38106]: pgmap v1367: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:44:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:44:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:44:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:37 smithi136 ceph-mon[29560]: pgmap v1367: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:37 smithi164 ceph-mon[33554]: pgmap v1367: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:39 smithi164 ceph-mon[33554]: pgmap v1368: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:39 smithi136 ceph-mon[29560]: pgmap v1368: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:39 smithi136 ceph-mon[38106]: pgmap v1368: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:41 smithi164 ceph-mon[33554]: pgmap v1369: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:41 smithi136 ceph-mon[29560]: pgmap v1369: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:41 smithi136 ceph-mon[38106]: pgmap v1369: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:43.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:44:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:44:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:43 smithi164 ceph-mon[33554]: pgmap v1370: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:43 smithi136 ceph-mon[29560]: pgmap v1370: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:43 smithi136 ceph-mon[38106]: pgmap v1370: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:44.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:44.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:45 smithi136 ceph-mon[38106]: pgmap v1371: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:45 smithi136 ceph-mon[29560]: pgmap v1371: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:45 smithi164 ceph-mon[33554]: pgmap v1371: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:47 smithi136 ceph-mon[38106]: pgmap v1372: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:47.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:47 smithi136 ceph-mon[29560]: pgmap v1372: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:44:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:44:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:44:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:47 smithi164 ceph-mon[33554]: pgmap v1372: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:49 smithi136 ceph-mon[38106]: pgmap v1373: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:49.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:49 smithi136 ceph-mon[29560]: pgmap v1373: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:49 smithi164 ceph-mon[33554]: pgmap v1373: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:51 smithi136 ceph-mon[38106]: pgmap v1374: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:51.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:51 smithi136 ceph-mon[29560]: pgmap v1374: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:51 smithi164 ceph-mon[33554]: pgmap v1374: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:53.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:44:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:44:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:53 smithi136 ceph-mon[38106]: pgmap v1375: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:53 smithi136 ceph-mon[29560]: pgmap v1375: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:53 smithi164 ceph-mon[33554]: pgmap v1375: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:54.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:54.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:44:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:55 smithi136 ceph-mon[29560]: pgmap v1376: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:55 smithi136 ceph-mon[38106]: pgmap v1376: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:55 smithi164 ceph-mon[33554]: pgmap v1376: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:57.499 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:44:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:44:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:44:57.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:57 smithi136 ceph-mon[38106]: pgmap v1377: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:57.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:57 smithi136 ceph-mon[29560]: pgmap v1377: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:57 smithi164 ceph-mon[33554]: pgmap v1377: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:44:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:44:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:44:59 smithi136 ceph-mon[38106]: pgmap v1378: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:44:59 smithi136 ceph-mon[29560]: pgmap v1378: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:44:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:44:59 smithi164 ceph-mon[33554]: pgmap v1378: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:01 smithi136 ceph-mon[38106]: pgmap v1379: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:01 smithi136 ceph-mon[29560]: pgmap v1379: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:01 smithi164 ceph-mon[33554]: pgmap v1379: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:03.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:45:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:45:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:03 smithi136 ceph-mon[38106]: pgmap v1380: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:03 smithi136 ceph-mon[29560]: pgmap v1380: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:03 smithi164 ceph-mon[33554]: pgmap v1380: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:05 smithi136 ceph-mon[38106]: pgmap v1381: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:05 smithi136 ceph-mon[29560]: pgmap v1381: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:05 smithi164 ceph-mon[33554]: pgmap v1381: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:07.523 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:45:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:45:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:45:07.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:07 smithi136 ceph-mon[38106]: pgmap v1382: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:07.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:07 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:45:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:07 smithi136 ceph-mon[29560]: pgmap v1382: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:07 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:45:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:07 smithi164 ceph-mon[33554]: pgmap v1382: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:07 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:45:09.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:09 smithi164 ceph-mon[33554]: pgmap v1383: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:09.787 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:09 smithi136 ceph-mon[38106]: pgmap v1383: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:09.787 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:09 smithi136 ceph-mon[29560]: pgmap v1383: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:11 smithi136 ceph-mon[38106]: pgmap v1384: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:11.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:11 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:11.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:11 smithi136 ceph-mon[29560]: pgmap v1384: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:11.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:11.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:11 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:11.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:11 smithi164 ceph-mon[33554]: pgmap v1384: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:11.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:11.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:11 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:45:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:45:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:12 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:12 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:45:13.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:12 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:45:13.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:12 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:45:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:45:13.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:13.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:45:13.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:45:13.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:12 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:45:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:13 smithi164 ceph-mon[33554]: pgmap v1385: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:13 smithi136 ceph-mon[29560]: pgmap v1385: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:13 smithi136 ceph-mon[38106]: pgmap v1385: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:15 smithi164 ceph-mon[33554]: pgmap v1386: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:15 smithi136 ceph-mon[38106]: pgmap v1386: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:16.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:15 smithi136 ceph-mon[29560]: pgmap v1386: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:45:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:45:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:45:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:17 smithi164 ceph-mon[33554]: pgmap v1387: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:17 smithi136 ceph-mon[38106]: pgmap v1387: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:18.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:17 smithi136 ceph-mon[29560]: pgmap v1387: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:19 smithi164 ceph-mon[33554]: pgmap v1388: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:20.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:19 smithi136 ceph-mon[29560]: pgmap v1388: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:19 smithi136 ceph-mon[38106]: pgmap v1388: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:21 smithi164 ceph-mon[33554]: pgmap v1389: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:21 smithi136 ceph-mon[38106]: pgmap v1389: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:21 smithi136 ceph-mon[29560]: pgmap v1389: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:23.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:45:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:45:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:23 smithi164 ceph-mon[33554]: pgmap v1390: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:23 smithi136 ceph-mon[38106]: pgmap v1390: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:23 smithi136 ceph-mon[29560]: pgmap v1390: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:25 smithi164 ceph-mon[33554]: pgmap v1391: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:25 smithi136 ceph-mon[38106]: pgmap v1391: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:25 smithi136 ceph-mon[29560]: pgmap v1391: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:45:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:45:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:45:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:27 smithi164 ceph-mon[33554]: pgmap v1392: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:27 smithi136 ceph-mon[38106]: pgmap v1392: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:27 smithi136 ceph-mon[29560]: pgmap v1392: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:29 smithi164 ceph-mon[33554]: pgmap v1393: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:30.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:29 smithi136 ceph-mon[29560]: pgmap v1393: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:29 smithi136 ceph-mon[38106]: pgmap v1393: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:31 smithi164 ceph-mon[33554]: pgmap v1394: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:32.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:31 smithi136 ceph-mon[29560]: pgmap v1394: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:32.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:31 smithi136 ceph-mon[38106]: pgmap v1394: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:33.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:45:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:45:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:33 smithi164 ceph-mon[33554]: pgmap v1395: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:33 smithi136 ceph-mon[29560]: pgmap v1395: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:33 smithi136 ceph-mon[38106]: pgmap v1395: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:35 smithi164 ceph-mon[33554]: pgmap v1396: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:35 smithi136 ceph-mon[29560]: pgmap v1396: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:35 smithi136 ceph-mon[38106]: pgmap v1396: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:45:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:45:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:45:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:37 smithi164 ceph-mon[33554]: pgmap v1397: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:37 smithi136 ceph-mon[29560]: pgmap v1397: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:37 smithi136 ceph-mon[38106]: pgmap v1397: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:39 smithi164 ceph-mon[33554]: pgmap v1398: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:39 smithi136 ceph-mon[29560]: pgmap v1398: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:39 smithi136 ceph-mon[38106]: pgmap v1398: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:41 smithi164 ceph-mon[33554]: pgmap v1399: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:41 smithi136 ceph-mon[38106]: pgmap v1399: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:41 smithi136 ceph-mon[29560]: pgmap v1399: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:43.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:45:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:45:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:43 smithi164 ceph-mon[33554]: pgmap v1400: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:43 smithi136 ceph-mon[29560]: pgmap v1400: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:43 smithi136 ceph-mon[38106]: pgmap v1400: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:45.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:45 smithi164 ceph-mon[33554]: pgmap v1401: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:46.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:45 smithi136 ceph-mon[29560]: pgmap v1401: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:45 smithi136 ceph-mon[38106]: pgmap v1401: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:45:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:45:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:45:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:47 smithi164 ceph-mon[33554]: pgmap v1402: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:47 smithi136 ceph-mon[38106]: pgmap v1402: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:48.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:47 smithi136 ceph-mon[29560]: pgmap v1402: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:49 smithi136 ceph-mon[38106]: pgmap v1403: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:49 smithi136 ceph-mon[29560]: pgmap v1403: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:49 smithi164 ceph-mon[33554]: pgmap v1403: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:51 smithi136 ceph-mon[38106]: pgmap v1404: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:51 smithi136 ceph-mon[29560]: pgmap v1404: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:52.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:51 smithi164 ceph-mon[33554]: pgmap v1404: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:45:53.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:45:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:45:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:53 smithi136 ceph-mon[38106]: pgmap v1405: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:53 smithi136 ceph-mon[29560]: pgmap v1405: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:53 smithi164 ceph-mon[33554]: pgmap v1405: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:55.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:55.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:45:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:55 smithi136 ceph-mon[38106]: pgmap v1406: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:55 smithi136 ceph-mon[29560]: pgmap v1406: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:55 smithi164 ceph-mon[33554]: pgmap v1406: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:45:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:45:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:45:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:45:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:57 smithi136 ceph-mon[29560]: pgmap v1407: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:57 smithi136 ceph-mon[38106]: pgmap v1407: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:57 smithi164 ceph-mon[33554]: pgmap v1407: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:45:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:45:59 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:45:59 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:45:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:45:59 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:00.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:00 smithi136 ceph-mon[38106]: pgmap v1408: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:00.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:00 smithi136 ceph-mon[29560]: pgmap v1408: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:00.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:00 smithi164 ceph-mon[33554]: pgmap v1408: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:02 smithi136 ceph-mon[38106]: pgmap v1409: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:02.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:02 smithi136 ceph-mon[29560]: pgmap v1409: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:02.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:02 smithi164 ceph-mon[33554]: pgmap v1409: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:03.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:46:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:46:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:04 smithi136 ceph-mon[38106]: pgmap v1410: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:04 smithi136 ceph-mon[29560]: pgmap v1410: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:04 smithi164 ceph-mon[33554]: pgmap v1410: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:05 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:05 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:05 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:06 smithi136 ceph-mon[38106]: pgmap v1411: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:06 smithi136 ceph-mon[29560]: pgmap v1411: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:06 smithi164 ceph-mon[33554]: pgmap v1411: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:07.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:07 smithi136 ceph-mon[38106]: pgmap v1412: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:07.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:07 smithi136 ceph-mon[29560]: pgmap v1412: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:07.325 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:46:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:46:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:46:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:07 smithi164 ceph-mon[33554]: pgmap v1412: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:09.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:09 smithi164 ceph-mon[33554]: pgmap v1413: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:09 smithi136 ceph-mon[38106]: pgmap v1413: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:09 smithi136 ceph-mon[29560]: pgmap v1413: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:11 smithi164 ceph-mon[33554]: pgmap v1414: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:11 smithi136 ceph-mon[38106]: pgmap v1414: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:11 smithi136 ceph-mon[29560]: pgmap v1414: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:13.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:46:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:46:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:13 smithi164 ceph-mon[33554]: pgmap v1415: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:13.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:46:13.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:13.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:46:13.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:46:13.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:13 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:46:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[38106]: pgmap v1415: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:46:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:46:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:46:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:46:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[29560]: pgmap v1415: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:46:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:46:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:46:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:13 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:46:14.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:14.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:14.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:15 smithi164 ceph-mon[33554]: pgmap v1416: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:15 smithi136 ceph-mon[38106]: pgmap v1416: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:15.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:15 smithi136 ceph-mon[29560]: pgmap v1416: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:17 smithi136 ceph-mon[38106]: pgmap v1417: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:17.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:17 smithi136 ceph-mon[29560]: pgmap v1417: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:17.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:46:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:46:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:46:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:17 smithi164 ceph-mon[33554]: pgmap v1417: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:19 smithi164 ceph-mon[33554]: pgmap v1418: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:19 smithi136 ceph-mon[38106]: pgmap v1418: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:19 smithi136 ceph-mon[29560]: pgmap v1418: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:21 smithi164 ceph-mon[33554]: pgmap v1419: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:21 smithi136 ceph-mon[38106]: pgmap v1419: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:21 smithi136 ceph-mon[29560]: pgmap v1419: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:23.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:23 smithi164 ceph-mon[33554]: pgmap v1420: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:23.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:46:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:46:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:23 smithi136 ceph-mon[38106]: pgmap v1420: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:23 smithi136 ceph-mon[29560]: pgmap v1420: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:24.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:25 smithi164 ceph-mon[33554]: pgmap v1421: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:25 smithi136 ceph-mon[38106]: pgmap v1421: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:25 smithi136 ceph-mon[29560]: pgmap v1421: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:27 smithi136 ceph-mon[38106]: pgmap v1422: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:27 smithi136 ceph-mon[29560]: pgmap v1422: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:27.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:46:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:46:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:46:27.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:27 smithi164 ceph-mon[33554]: pgmap v1422: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:28.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:29 smithi164 ceph-mon[33554]: pgmap v1423: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:29 smithi136 ceph-mon[38106]: pgmap v1423: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:29 smithi136 ceph-mon[29560]: pgmap v1423: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:31 smithi164 ceph-mon[33554]: pgmap v1424: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:31 smithi136 ceph-mon[38106]: pgmap v1424: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:31.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:31 smithi136 ceph-mon[29560]: pgmap v1424: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:33.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:46:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:46:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:33 smithi164 ceph-mon[33554]: pgmap v1425: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:33 smithi136 ceph-mon[38106]: pgmap v1425: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:33 smithi136 ceph-mon[29560]: pgmap v1425: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:34.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:35.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:35 smithi164 ceph-mon[33554]: pgmap v1426: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:35 smithi136 ceph-mon[38106]: pgmap v1426: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:35 smithi136 ceph-mon[29560]: pgmap v1426: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:37 smithi136 ceph-mon[38106]: pgmap v1427: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:37 smithi136 ceph-mon[29560]: pgmap v1427: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:46:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:46:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:46:37.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:37 smithi164 ceph-mon[33554]: pgmap v1427: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:39.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:39 smithi164 ceph-mon[33554]: pgmap v1428: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:39 smithi136 ceph-mon[38106]: pgmap v1428: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:39 smithi136 ceph-mon[29560]: pgmap v1428: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:41 smithi164 ceph-mon[33554]: pgmap v1429: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:41 smithi136 ceph-mon[38106]: pgmap v1429: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:41 smithi136 ceph-mon[29560]: pgmap v1429: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:43.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:46:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:46:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:43 smithi164 ceph-mon[33554]: pgmap v1430: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:43 smithi136 ceph-mon[29560]: pgmap v1430: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:43 smithi136 ceph-mon[38106]: pgmap v1430: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:44.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:44.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:45.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:45 smithi164 ceph-mon[33554]: pgmap v1431: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:45 smithi136 ceph-mon[38106]: pgmap v1431: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:45 smithi136 ceph-mon[29560]: pgmap v1431: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:46:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:46:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:46:47.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:47 smithi136 ceph-mon[38106]: pgmap v1432: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:47.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:47 smithi136 ceph-mon[29560]: pgmap v1432: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:47.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:47 smithi164 ceph-mon[33554]: pgmap v1432: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:49 smithi164 ceph-mon[33554]: pgmap v1433: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:49 smithi136 ceph-mon[38106]: pgmap v1433: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:49.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:49 smithi136 ceph-mon[29560]: pgmap v1433: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:51.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:51 smithi164 ceph-mon[33554]: pgmap v1434: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:51 smithi136 ceph-mon[38106]: pgmap v1434: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:51.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:51 smithi136 ceph-mon[29560]: pgmap v1434: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:53.432 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:46:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:46:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:53 smithi164 ceph-mon[33554]: pgmap v1435: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:53 smithi136 ceph-mon[38106]: pgmap v1435: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:53 smithi136 ceph-mon[29560]: pgmap v1435: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:54.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:54.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:54.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:46:55.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:55 smithi164 ceph-mon[33554]: pgmap v1436: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:55 smithi136 ceph-mon[38106]: pgmap v1436: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:55 smithi136 ceph-mon[29560]: pgmap v1436: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:57.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:57 smithi136 ceph-mon[29560]: pgmap v1437: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:57.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:57 smithi136 ceph-mon[38106]: pgmap v1437: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:46:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:46:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:46:57.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:57 smithi164 ceph-mon[33554]: pgmap v1437: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:46:58.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:46:59.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:46:59 smithi164 ceph-mon[33554]: pgmap v1438: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:46:59 smithi136 ceph-mon[38106]: pgmap v1438: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:46:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:46:59 smithi136 ceph-mon[29560]: pgmap v1438: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:01.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:01 smithi164 ceph-mon[33554]: pgmap v1439: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:01 smithi136 ceph-mon[38106]: pgmap v1439: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:01 smithi136 ceph-mon[29560]: pgmap v1439: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:03.462 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:47:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:47:03.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:03 smithi164 ceph-mon[33554]: pgmap v1440: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:03 smithi136 ceph-mon[38106]: pgmap v1440: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:03 smithi136 ceph-mon[29560]: pgmap v1440: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:05 smithi136 ceph-mon[38106]: pgmap v1441: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:05 smithi136 ceph-mon[29560]: pgmap v1441: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:05 smithi164 ceph-mon[33554]: pgmap v1441: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:07.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:07 smithi136 ceph-mon[38106]: pgmap v1442: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:07.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:07 smithi136 ceph-mon[29560]: pgmap v1442: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:07.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:47:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:47:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:47:07.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:07 smithi164 ceph-mon[33554]: pgmap v1442: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:09 smithi136 ceph-mon[38106]: pgmap v1443: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:09 smithi136 ceph-mon[29560]: pgmap v1443: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:09 smithi164 ceph-mon[33554]: pgmap v1443: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:11 smithi136 ceph-mon[38106]: pgmap v1444: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:11 smithi136 ceph-mon[29560]: pgmap v1444: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:11 smithi164 ceph-mon[33554]: pgmap v1444: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:13.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:47:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:47:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:13 smithi136 ceph-mon[38106]: pgmap v1445: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:47:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:13 smithi136 ceph-mon[29560]: pgmap v1445: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:47:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:13 smithi164 ceph-mon[33554]: pgmap v1445: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:47:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:47:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:47:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:47:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:47:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:14 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:14 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:14 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:14 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:47:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:14 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:47:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:14 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:47:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:15 smithi136 ceph-mon[38106]: pgmap v1446: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:15 smithi136 ceph-mon[29560]: pgmap v1446: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:15 smithi164 ceph-mon[33554]: pgmap v1446: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:47:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:47:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:47:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:17 smithi136 ceph-mon[38106]: pgmap v1447: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:17 smithi136 ceph-mon[29560]: pgmap v1447: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:17 smithi164 ceph-mon[33554]: pgmap v1447: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:19 smithi136 ceph-mon[38106]: pgmap v1448: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:20.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:19 smithi136 ceph-mon[29560]: pgmap v1448: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:19 smithi164 ceph-mon[33554]: pgmap v1448: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:21 smithi136 ceph-mon[38106]: pgmap v1449: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:21 smithi136 ceph-mon[29560]: pgmap v1449: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:21 smithi164 ceph-mon[33554]: pgmap v1449: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:23.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:47:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:47:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:23 smithi136 ceph-mon[38106]: pgmap v1450: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:23 smithi136 ceph-mon[29560]: pgmap v1450: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:23 smithi164 ceph-mon[33554]: pgmap v1450: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:25 smithi164 ceph-mon[33554]: pgmap v1451: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:25 smithi136 ceph-mon[38106]: pgmap v1451: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:25 smithi136 ceph-mon[29560]: pgmap v1451: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:47:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:47:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:47:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:27 smithi164 ceph-mon[33554]: pgmap v1452: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:27 smithi136 ceph-mon[38106]: pgmap v1452: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:27 smithi136 ceph-mon[29560]: pgmap v1452: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:29 smithi164 ceph-mon[33554]: pgmap v1453: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:29 smithi136 ceph-mon[38106]: pgmap v1453: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:30.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:29 smithi136 ceph-mon[29560]: pgmap v1453: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:31 smithi164 ceph-mon[33554]: pgmap v1454: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:31 smithi136 ceph-mon[38106]: pgmap v1454: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:31 smithi136 ceph-mon[29560]: pgmap v1454: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:33.470 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:47:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:47:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:33 smithi164 ceph-mon[33554]: pgmap v1455: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:33 smithi136 ceph-mon[38106]: pgmap v1455: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:33 smithi136 ceph-mon[29560]: pgmap v1455: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:35 smithi164 ceph-mon[33554]: pgmap v1456: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:35 smithi136 ceph-mon[29560]: pgmap v1456: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:35 smithi136 ceph-mon[38106]: pgmap v1456: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:47:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:47:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:47:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:37 smithi164 ceph-mon[33554]: pgmap v1457: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:37 smithi136 ceph-mon[38106]: pgmap v1457: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:37 smithi136 ceph-mon[29560]: pgmap v1457: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:39 smithi164 ceph-mon[33554]: pgmap v1458: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:39 smithi136 ceph-mon[38106]: pgmap v1458: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:39 smithi136 ceph-mon[29560]: pgmap v1458: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:41 smithi164 ceph-mon[33554]: pgmap v1459: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:41 smithi136 ceph-mon[38106]: pgmap v1459: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:41 smithi136 ceph-mon[29560]: pgmap v1459: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:43.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:47:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:47:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:43 smithi164 ceph-mon[33554]: pgmap v1460: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:43 smithi136 ceph-mon[38106]: pgmap v1460: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:43 smithi136 ceph-mon[29560]: pgmap v1460: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:45.225 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:45.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:45 smithi164 ceph-mon[33554]: pgmap v1461: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:45 smithi136 ceph-mon[38106]: pgmap v1461: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:46.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:45 smithi136 ceph-mon[29560]: pgmap v1461: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:47.578 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:47:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:47:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:47:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:47 smithi164 ceph-mon[33554]: pgmap v1462: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:47 smithi136 ceph-mon[38106]: pgmap v1462: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:48.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:47 smithi136 ceph-mon[29560]: pgmap v1462: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:49 smithi164 ceph-mon[33554]: pgmap v1463: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:49 smithi136 ceph-mon[38106]: pgmap v1463: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:49 smithi136 ceph-mon[29560]: pgmap v1463: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:51 smithi164 ceph-mon[33554]: pgmap v1464: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:51 smithi136 ceph-mon[29560]: pgmap v1464: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:51 smithi136 ceph-mon[38106]: pgmap v1464: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:47:53.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:47:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:47:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:53 smithi164 ceph-mon[33554]: pgmap v1465: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:53 smithi136 ceph-mon[29560]: pgmap v1465: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:53 smithi136 ceph-mon[38106]: pgmap v1465: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:55.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:55.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:47:56.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:55 smithi164 ceph-mon[33554]: pgmap v1466: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:55 smithi136 ceph-mon[38106]: pgmap v1466: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:55 smithi136 ceph-mon[29560]: pgmap v1466: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:47:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:47:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:47:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:47:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:57 smithi164 ceph-mon[33554]: pgmap v1467: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:57 smithi136 ceph-mon[38106]: pgmap v1467: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:57 smithi136 ceph-mon[29560]: pgmap v1467: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:47:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:47:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:47:59 smithi164 ceph-mon[33554]: pgmap v1468: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:00.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:47:59 smithi136 ceph-mon[38106]: pgmap v1468: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:00.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:47:59 smithi136 ceph-mon[29560]: pgmap v1468: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:02.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:01 smithi164 ceph-mon[33554]: pgmap v1469: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:01 smithi136 ceph-mon[38106]: pgmap v1469: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:02.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:01 smithi136 ceph-mon[29560]: pgmap v1469: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:03.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:48:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:48:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:03 smithi164 ceph-mon[33554]: pgmap v1470: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:03 smithi136 ceph-mon[38106]: pgmap v1470: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:03 smithi136 ceph-mon[29560]: pgmap v1470: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:05.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:06.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:05 smithi164 ceph-mon[33554]: pgmap v1471: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:05 smithi136 ceph-mon[38106]: pgmap v1471: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:05 smithi136 ceph-mon[29560]: pgmap v1471: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:48:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:48:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:48:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:07 smithi164 ceph-mon[33554]: pgmap v1472: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:08.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:07 smithi136 ceph-mon[38106]: pgmap v1472: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:08.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:07 smithi136 ceph-mon[29560]: pgmap v1472: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:09 smithi164 ceph-mon[33554]: pgmap v1473: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:09 smithi136 ceph-mon[29560]: pgmap v1473: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:09 smithi136 ceph-mon[38106]: pgmap v1473: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:11 smithi164 ceph-mon[33554]: pgmap v1474: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:11 smithi136 ceph-mon[38106]: pgmap v1474: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:11 smithi136 ceph-mon[29560]: pgmap v1474: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:13.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:48:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:48:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:13 smithi164 ceph-mon[33554]: pgmap v1475: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:13 smithi136 ceph-mon[38106]: pgmap v1475: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:13 smithi136 ceph-mon[29560]: pgmap v1475: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:14 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:48:15.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:15.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:14 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:48:15.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:15.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:14 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:48:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:15 smithi164 ceph-mon[33554]: pgmap v1476: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:15 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:48:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:15 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:48:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:15 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[38106]: pgmap v1476: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:48:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:48:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[29560]: pgmap v1476: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:48:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:48:16.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:15 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:48:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:48:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:48:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:48:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:17 smithi164 ceph-mon[33554]: pgmap v1477: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:17 smithi136 ceph-mon[38106]: pgmap v1477: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:18.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:17 smithi136 ceph-mon[29560]: pgmap v1477: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:19 smithi164 ceph-mon[33554]: pgmap v1478: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:19 smithi136 ceph-mon[38106]: pgmap v1478: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:20.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:19 smithi136 ceph-mon[29560]: pgmap v1478: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:21 smithi164 ceph-mon[33554]: pgmap v1479: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:21 smithi136 ceph-mon[38106]: pgmap v1479: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:21 smithi136 ceph-mon[29560]: pgmap v1479: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:23.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:48:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:48:24.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:23 smithi164 ceph-mon[33554]: pgmap v1480: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:23 smithi136 ceph-mon[38106]: pgmap v1480: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:23 smithi136 ceph-mon[29560]: pgmap v1480: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:25 smithi164 ceph-mon[33554]: pgmap v1481: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:25 smithi136 ceph-mon[38106]: pgmap v1481: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:25 smithi136 ceph-mon[29560]: pgmap v1481: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:48:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:48:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:48:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:27 smithi164 ceph-mon[33554]: pgmap v1482: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:27 smithi136 ceph-mon[38106]: pgmap v1482: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:27 smithi136 ceph-mon[29560]: pgmap v1482: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:29 smithi164 ceph-mon[33554]: pgmap v1483: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:29 smithi136 ceph-mon[38106]: pgmap v1483: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:30.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:29 smithi136 ceph-mon[29560]: pgmap v1483: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:31 smithi164 ceph-mon[33554]: pgmap v1484: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:31 smithi136 ceph-mon[38106]: pgmap v1484: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:31 smithi136 ceph-mon[29560]: pgmap v1484: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:33.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:48:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:48:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:33 smithi164 ceph-mon[33554]: pgmap v1485: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:33 smithi136 ceph-mon[38106]: pgmap v1485: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:33 smithi136 ceph-mon[29560]: pgmap v1485: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:35 smithi136 ceph-mon[38106]: pgmap v1486: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:35 smithi136 ceph-mon[29560]: pgmap v1486: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:36.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:35 smithi164 ceph-mon[33554]: pgmap v1486: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:48:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:48:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:48:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:37 smithi136 ceph-mon[38106]: pgmap v1487: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:37 smithi136 ceph-mon[29560]: pgmap v1487: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:38.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:37 smithi164 ceph-mon[33554]: pgmap v1487: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:39 smithi136 ceph-mon[38106]: pgmap v1488: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:39 smithi136 ceph-mon[29560]: pgmap v1488: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:40.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:39 smithi164 ceph-mon[33554]: pgmap v1488: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:48:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:41 smithi136 ceph-mon[38106]: pgmap v1489: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:41 smithi136 ceph-mon[29560]: pgmap v1489: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:42.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:41 smithi164 ceph-mon[33554]: pgmap v1489: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:43.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:48:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:48:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:43 smithi136 ceph-mon[38106]: pgmap v1490: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:43 smithi136 ceph-mon[29560]: pgmap v1490: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:43 smithi164 ceph-mon[33554]: pgmap v1490: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:44.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:45.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:45.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:45.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:45 smithi136 ceph-mon[38106]: pgmap v1491: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:46.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:45 smithi136 ceph-mon[29560]: pgmap v1491: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:46.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:45 smithi164 ceph-mon[33554]: pgmap v1491: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:48:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:48:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:48:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:48 smithi136 ceph-mon[38106]: pgmap v1492: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:48.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:48 smithi136 ceph-mon[29560]: pgmap v1492: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:48.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:48 smithi164 ceph-mon[33554]: pgmap v1492: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:50 smithi136 ceph-mon[38106]: pgmap v1493: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:50 smithi136 ceph-mon[29560]: pgmap v1493: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:50.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:50 smithi164 ceph-mon[33554]: pgmap v1493: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:52 smithi136 ceph-mon[38106]: pgmap v1494: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:52 smithi136 ceph-mon[29560]: pgmap v1494: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:52.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:52 smithi164 ceph-mon[33554]: pgmap v1494: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:53.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:48:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:48:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:54 smithi136 ceph-mon[38106]: pgmap v1495: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:54 smithi136 ceph-mon[29560]: pgmap v1495: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:54 smithi164 ceph-mon[33554]: pgmap v1495: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:55.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:55 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:55.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:55 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:55 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:48:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:56 smithi136 ceph-mon[38106]: pgmap v1496: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:56 smithi136 ceph-mon[29560]: pgmap v1496: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:56 smithi164 ceph-mon[33554]: pgmap v1496: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:48:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:48:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:48:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:48:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:58 smithi136 ceph-mon[38106]: pgmap v1497: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:58 smithi136 ceph-mon[29560]: pgmap v1497: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:58 smithi164 ceph-mon[33554]: pgmap v1497: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:48:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:48:59 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:48:59 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:48:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:48:59 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:00.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:00 smithi136 ceph-mon[38106]: pgmap v1498: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:00.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:00 smithi136 ceph-mon[29560]: pgmap v1498: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:00.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:00 smithi164 ceph-mon[33554]: pgmap v1498: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:02 smithi136 ceph-mon[38106]: pgmap v1499: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:02.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:02 smithi136 ceph-mon[29560]: pgmap v1499: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:02 smithi164 ceph-mon[33554]: pgmap v1499: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:03.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:49:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:49:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:04 smithi136 ceph-mon[38106]: pgmap v1500: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:04 smithi136 ceph-mon[29560]: pgmap v1500: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:04 smithi164 ceph-mon[33554]: pgmap v1500: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:05 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:05 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:05 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:06 smithi136 ceph-mon[38106]: pgmap v1501: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:06 smithi136 ceph-mon[29560]: pgmap v1501: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:06 smithi164 ceph-mon[33554]: pgmap v1501: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:07.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:07 smithi136 ceph-mon[38106]: pgmap v1502: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:07.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:07 smithi136 ceph-mon[29560]: pgmap v1502: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:07.325 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:49:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:49:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:49:07.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:07 smithi164 ceph-mon[33554]: pgmap v1502: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:09.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:09 smithi164 ceph-mon[33554]: pgmap v1503: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:09 smithi136 ceph-mon[38106]: pgmap v1503: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:09 smithi136 ceph-mon[29560]: pgmap v1503: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:11.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:11 smithi164 ceph-mon[33554]: pgmap v1504: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:11 smithi136 ceph-mon[38106]: pgmap v1504: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:11 smithi136 ceph-mon[29560]: pgmap v1504: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:13 smithi164 ceph-mon[33554]: pgmap v1505: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:13.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:13.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:49:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:49:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:13 smithi136 ceph-mon[38106]: pgmap v1505: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:13 smithi136 ceph-mon[29560]: pgmap v1505: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:14.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:14.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:14.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:15.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:15 smithi164 ceph-mon[33554]: pgmap v1506: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:15.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:15 smithi136 ceph-mon[38106]: pgmap v1506: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:15.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:15 smithi136 ceph-mon[29560]: pgmap v1506: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:16.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:16 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:49:16.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:16 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:49:16.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:16 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:49:16.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:16 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:49:16.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:16 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:49:16.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:16 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:49:16.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:16 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:49:16.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:16 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:49:16.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:16 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:49:16.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:16 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:49:16.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:16 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:49:16.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:16 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:49:17.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:17 smithi136 ceph-mon[38106]: pgmap v1507: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:49:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:49:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:49:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:17 smithi136 ceph-mon[29560]: pgmap v1507: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:17.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:17 smithi164 ceph-mon[33554]: pgmap v1507: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:19.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:19 smithi164 ceph-mon[33554]: pgmap v1508: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:19 smithi136 ceph-mon[38106]: pgmap v1508: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:19.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:19 smithi136 ceph-mon[29560]: pgmap v1508: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:21 smithi164 ceph-mon[33554]: pgmap v1509: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:21.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:21 smithi136 ceph-mon[38106]: pgmap v1509: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:21.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:21 smithi136 ceph-mon[29560]: pgmap v1509: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:23.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:23 smithi164 ceph-mon[33554]: pgmap v1510: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:23.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:49:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:49:23.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:23 smithi136 ceph-mon[38106]: pgmap v1510: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:23.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:23 smithi136 ceph-mon[29560]: pgmap v1510: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:24.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:25.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:25 smithi164 ceph-mon[33554]: pgmap v1511: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:25 smithi136 ceph-mon[38106]: pgmap v1511: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:25.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:25 smithi136 ceph-mon[29560]: pgmap v1511: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:27 smithi136 ceph-mon[38106]: pgmap v1512: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:49:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:49:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:49:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:27 smithi136 ceph-mon[29560]: pgmap v1512: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:27.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:27 smithi164 ceph-mon[33554]: pgmap v1512: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:28.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:28.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:28.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:29 smithi164 ceph-mon[33554]: pgmap v1513: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:29 smithi136 ceph-mon[38106]: pgmap v1513: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:29.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:29 smithi136 ceph-mon[29560]: pgmap v1513: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:31.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:31 smithi164 ceph-mon[33554]: pgmap v1514: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:31 smithi136 ceph-mon[38106]: pgmap v1514: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:31.856 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:31 smithi136 ceph-mon[29560]: pgmap v1514: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:33 smithi164 ceph-mon[33554]: pgmap v1515: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:33.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:49:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:49:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:33 smithi136 ceph-mon[38106]: pgmap v1515: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:33.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:33 smithi136 ceph-mon[29560]: pgmap v1515: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:34.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:34.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:35 smithi136 ceph-mon[38106]: pgmap v1516: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:35 smithi136 ceph-mon[29560]: pgmap v1516: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:35 smithi164 ceph-mon[33554]: pgmap v1516: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:37.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:37 smithi136 ceph-mon[29560]: pgmap v1517: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:37.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:37 smithi136 ceph-mon[38106]: pgmap v1517: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:49:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:49:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:49:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:37 smithi164 ceph-mon[33554]: pgmap v1517: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:39 smithi136 ceph-mon[38106]: pgmap v1518: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:39 smithi136 ceph-mon[29560]: pgmap v1518: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:39 smithi164 ceph-mon[33554]: pgmap v1518: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:41 smithi136 ceph-mon[38106]: pgmap v1519: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:41 smithi136 ceph-mon[29560]: pgmap v1519: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:41 smithi164 ceph-mon[33554]: pgmap v1519: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:43 smithi164 ceph-mon[33554]: pgmap v1520: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:43.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:43.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:49:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:49:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:43 smithi136 ceph-mon[38106]: pgmap v1520: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:43 smithi136 ceph-mon[29560]: pgmap v1520: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:44.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:44.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:45 smithi136 ceph-mon[29560]: pgmap v1521: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:45 smithi136 ceph-mon[38106]: pgmap v1521: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:45 smithi164 ceph-mon[33554]: pgmap v1521: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:47.494 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:49:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:49:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:49:47.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:47 smithi136 ceph-mon[29560]: pgmap v1522: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:47 smithi136 ceph-mon[38106]: pgmap v1522: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:47 smithi164 ceph-mon[33554]: pgmap v1522: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:49 smithi136 ceph-mon[38106]: pgmap v1523: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:49.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:49 smithi136 ceph-mon[29560]: pgmap v1523: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:49 smithi164 ceph-mon[33554]: pgmap v1523: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:49:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:51 smithi136 ceph-mon[38106]: pgmap v1524: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:51.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:51 smithi136 ceph-mon[29560]: pgmap v1524: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:51 smithi164 ceph-mon[33554]: pgmap v1524: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:53.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:53 smithi164 ceph-mon[33554]: pgmap v1525: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:53.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:49:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:49:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:53 smithi136 ceph-mon[38106]: pgmap v1525: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:53 smithi136 ceph-mon[29560]: pgmap v1525: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:54.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:54.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:49:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:55 smithi136 ceph-mon[38106]: pgmap v1526: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:55 smithi136 ceph-mon[29560]: pgmap v1526: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:55 smithi164 ceph-mon[33554]: pgmap v1526: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:57.513 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:49:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:49:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:49:57.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:57 smithi136 ceph-mon[38106]: pgmap v1527: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:57.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:57 smithi136 ceph-mon[29560]: pgmap v1527: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:57 smithi164 ceph-mon[33554]: pgmap v1527: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:49:58.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:58.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:49:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:49:59 smithi136 ceph-mon[38106]: pgmap v1528: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:49:59 smithi136 ceph-mon[29560]: pgmap v1528: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:49:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:49:59 smithi164 ceph-mon[33554]: pgmap v1528: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:00.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:00 smithi136 ceph-mon[38106]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:50:00.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:00 smithi136 ceph-mon[29560]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:50:00.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:00 smithi164 ceph-mon[33554]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-26T05:50:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:01 smithi136 ceph-mon[38106]: pgmap v1529: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:01 smithi136 ceph-mon[29560]: pgmap v1529: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:01 smithi164 ceph-mon[33554]: pgmap v1529: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:03.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:03 smithi164 ceph-mon[33554]: pgmap v1530: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:03.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:50:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:50:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:03 smithi136 ceph-mon[38106]: pgmap v1530: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:03 smithi136 ceph-mon[29560]: pgmap v1530: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:05 smithi136 ceph-mon[38106]: pgmap v1531: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:05 smithi136 ceph-mon[29560]: pgmap v1531: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:05 smithi164 ceph-mon[33554]: pgmap v1531: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:07.542 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:50:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:50:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:50:07.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:07 smithi136 ceph-mon[38106]: pgmap v1532: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:07.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:07 smithi136 ceph-mon[29560]: pgmap v1532: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:07 smithi164 ceph-mon[33554]: pgmap v1532: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:09 smithi136 ceph-mon[38106]: pgmap v1533: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:09 smithi136 ceph-mon[29560]: pgmap v1533: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:09 smithi164 ceph-mon[33554]: pgmap v1533: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:11 smithi136 ceph-mon[38106]: pgmap v1534: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:11 smithi136 ceph-mon[29560]: pgmap v1534: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:11 smithi164 ceph-mon[33554]: pgmap v1534: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:13 smithi164 ceph-mon[33554]: pgmap v1535: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:13.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:13.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:50:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:50:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:13 smithi136 ceph-mon[38106]: pgmap v1535: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:13 smithi136 ceph-mon[29560]: pgmap v1535: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:13.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:14.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:14.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:15.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:15 smithi136 ceph-mon[38106]: pgmap v1536: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:15.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:15 smithi136 ceph-mon[29560]: pgmap v1536: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:15 smithi164 ceph-mon[33554]: pgmap v1536: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:16.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:16 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:50:16.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:16 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:50:16.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:16 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:50:17.570 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:50:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:50:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:50:17.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:17 smithi136 ceph-mon[38106]: pgmap v1537: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:17.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:17 smithi136 ceph-mon[29560]: pgmap v1537: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:17 smithi164 ceph-mon[33554]: pgmap v1537: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:19.593 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:19 smithi136 ceph-mon[38106]: pgmap v1538: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:19.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:19 smithi136 ceph-mon[29560]: pgmap v1538: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:19 smithi164 ceph-mon[33554]: pgmap v1538: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:21 smithi164 ceph-mon[33554]: pgmap v1539: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:21 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:50:21.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:21 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:50:21.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:21 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[38106]: pgmap v1539: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[29560]: pgmap v1539: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:22.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:50:22.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:50:22.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:21 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:50:23.617 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:50:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:50:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:23 smithi164 ceph-mon[33554]: pgmap v1540: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:23 smithi136 ceph-mon[38106]: pgmap v1540: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:23 smithi136 ceph-mon[29560]: pgmap v1540: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:25 smithi164 ceph-mon[33554]: pgmap v1541: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:25 smithi136 ceph-mon[38106]: pgmap v1541: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:25 smithi136 ceph-mon[29560]: pgmap v1541: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:50:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:50:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:50:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:27 smithi164 ceph-mon[33554]: pgmap v1542: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:27 smithi136 ceph-mon[38106]: pgmap v1542: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:27 smithi136 ceph-mon[29560]: pgmap v1542: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:29 smithi164 ceph-mon[33554]: pgmap v1543: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:30.031 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:29 smithi136 ceph-mon[38106]: pgmap v1543: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:30.031 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:29 smithi136 ceph-mon[29560]: pgmap v1543: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:31 smithi164 ceph-mon[33554]: pgmap v1544: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:31 smithi136 ceph-mon[38106]: pgmap v1544: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:31 smithi136 ceph-mon[29560]: pgmap v1544: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:33.646 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:50:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:50:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:33 smithi164 ceph-mon[33554]: pgmap v1545: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:33 smithi136 ceph-mon[29560]: pgmap v1545: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:33 smithi136 ceph-mon[38106]: pgmap v1545: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:35.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:35.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:35 smithi164 ceph-mon[33554]: pgmap v1546: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:35 smithi136 ceph-mon[38106]: pgmap v1546: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:35 smithi136 ceph-mon[29560]: pgmap v1546: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:50:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:50:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:50:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:37 smithi164 ceph-mon[33554]: pgmap v1547: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:37 smithi136 ceph-mon[29560]: pgmap v1547: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:37 smithi136 ceph-mon[38106]: pgmap v1547: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:39 smithi164 ceph-mon[33554]: pgmap v1548: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:39 smithi136 ceph-mon[38106]: pgmap v1548: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:39 smithi136 ceph-mon[29560]: pgmap v1548: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:41 smithi164 ceph-mon[33554]: pgmap v1549: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:41 smithi136 ceph-mon[38106]: pgmap v1549: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:41 smithi136 ceph-mon[29560]: pgmap v1549: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:43.669 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:50:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:50:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:43 smithi164 ceph-mon[33554]: pgmap v1550: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:43 smithi136 ceph-mon[38106]: pgmap v1550: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:43 smithi136 ceph-mon[29560]: pgmap v1550: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:45 smithi164 ceph-mon[33554]: pgmap v1551: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:45.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:45 smithi136 ceph-mon[38106]: pgmap v1551: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:45.976 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:45 smithi136 ceph-mon[29560]: pgmap v1551: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:50:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:50:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:50:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:47 smithi164 ceph-mon[33554]: pgmap v1552: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:47 smithi136 ceph-mon[38106]: pgmap v1552: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:47 smithi136 ceph-mon[29560]: pgmap v1552: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:49 smithi164 ceph-mon[33554]: pgmap v1553: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:50.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:49 smithi136 ceph-mon[38106]: pgmap v1553: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:50.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:49 smithi136 ceph-mon[29560]: pgmap v1553: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:50:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:51 smithi164 ceph-mon[33554]: pgmap v1554: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:51 smithi136 ceph-mon[29560]: pgmap v1554: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:51 smithi136 ceph-mon[38106]: pgmap v1554: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:53.687 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:50:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:50:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:53 smithi164 ceph-mon[33554]: pgmap v1555: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:53 smithi136 ceph-mon[38106]: pgmap v1555: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:53 smithi136 ceph-mon[29560]: pgmap v1555: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:50:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:55 smithi164 ceph-mon[33554]: pgmap v1556: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:55 smithi136 ceph-mon[38106]: pgmap v1556: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:55 smithi136 ceph-mon[29560]: pgmap v1556: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:50:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:50:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:50:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:50:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:57 smithi164 ceph-mon[33554]: pgmap v1557: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:57 smithi136 ceph-mon[29560]: pgmap v1557: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:57 smithi136 ceph-mon[38106]: pgmap v1557: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:50:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:50:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:50:59 smithi164 ceph-mon[33554]: pgmap v1558: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:50:59 smithi136 ceph-mon[29560]: pgmap v1558: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:50:59 smithi136 ceph-mon[38106]: pgmap v1558: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:01 smithi164 ceph-mon[33554]: pgmap v1559: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:01 smithi136 ceph-mon[38106]: pgmap v1559: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:02.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:01 smithi136 ceph-mon[29560]: pgmap v1559: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:03.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:51:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:51:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:03 smithi164 ceph-mon[33554]: pgmap v1560: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:03 smithi136 ceph-mon[38106]: pgmap v1560: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:03 smithi136 ceph-mon[29560]: pgmap v1560: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:05.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:05.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:05 smithi164 ceph-mon[33554]: pgmap v1561: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:05 smithi136 ceph-mon[38106]: pgmap v1561: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:06.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:05 smithi136 ceph-mon[29560]: pgmap v1561: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:51:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:51:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:51:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:07 smithi136 ceph-mon[38106]: pgmap v1562: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:08.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:07 smithi136 ceph-mon[29560]: pgmap v1562: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:08.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:07 smithi164 ceph-mon[33554]: pgmap v1562: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:09 smithi136 ceph-mon[38106]: pgmap v1563: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:09 smithi136 ceph-mon[29560]: pgmap v1563: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:10.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:09 smithi164 ceph-mon[33554]: pgmap v1563: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:11 smithi136 ceph-mon[38106]: pgmap v1564: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:11 smithi136 ceph-mon[29560]: pgmap v1564: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:11 smithi164 ceph-mon[33554]: pgmap v1564: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:13.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:51:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:51:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:13 smithi136 ceph-mon[38106]: pgmap v1565: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:13 smithi136 ceph-mon[29560]: pgmap v1565: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:13 smithi164 ceph-mon[33554]: pgmap v1565: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:14.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:15.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:15 smithi136 ceph-mon[38106]: pgmap v1566: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:16.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:15 smithi136 ceph-mon[29560]: pgmap v1566: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:15 smithi164 ceph-mon[33554]: pgmap v1566: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:51:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:51:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:51:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:17 smithi136 ceph-mon[38106]: pgmap v1567: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:17 smithi136 ceph-mon[29560]: pgmap v1567: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:17 smithi164 ceph-mon[33554]: pgmap v1567: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:19 smithi136 ceph-mon[38106]: pgmap v1568: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:20.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:19 smithi136 ceph-mon[29560]: pgmap v1568: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:19 smithi164 ceph-mon[33554]: pgmap v1568: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:20 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:51:21.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:20 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:51:21.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:20 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:51:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:21 smithi136 ceph-mon[38106]: pgmap v1569: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:21 smithi136 ceph-mon[29560]: pgmap v1569: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:22.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:21 smithi164 ceph-mon[33554]: pgmap v1569: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:23.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:51:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:51:24.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:23 smithi136 ceph-mon[38106]: pgmap v1570: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:24.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:23 smithi136 ceph-mon[29560]: pgmap v1570: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:24.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:23 smithi164 ceph-mon[33554]: pgmap v1570: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:25 smithi164 ceph-mon[33554]: pgmap v1571: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:25 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:25 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:25 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:51:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:25 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:51:26.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:25 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[38106]: pgmap v1571: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:51:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:51:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[29560]: pgmap v1571: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:26.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:26.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:51:26.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:51:26.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:25 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:51:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:51:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:51:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:51:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:27 smithi164 ceph-mon[33554]: pgmap v1572: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:28.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:27 smithi136 ceph-mon[29560]: pgmap v1572: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:28.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:27 smithi136 ceph-mon[38106]: pgmap v1572: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:29.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:29.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:29 smithi164 ceph-mon[33554]: pgmap v1573: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:29 smithi136 ceph-mon[38106]: pgmap v1573: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:30.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:29 smithi136 ceph-mon[29560]: pgmap v1573: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:32.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:31 smithi164 ceph-mon[33554]: pgmap v1574: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:32.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:31 smithi136 ceph-mon[38106]: pgmap v1574: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:32.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:31 smithi136 ceph-mon[29560]: pgmap v1574: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:33.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:51:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:51:34.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:33 smithi164 ceph-mon[33554]: pgmap v1575: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:33 smithi136 ceph-mon[38106]: pgmap v1575: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:33 smithi136 ceph-mon[29560]: pgmap v1575: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:35.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:35.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:35 smithi164 ceph-mon[33554]: pgmap v1576: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:35 smithi136 ceph-mon[29560]: pgmap v1576: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:35 smithi136 ceph-mon[38106]: pgmap v1576: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:51:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:51:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:51:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:37 smithi164 ceph-mon[33554]: pgmap v1577: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:37 smithi136 ceph-mon[29560]: pgmap v1577: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:37 smithi136 ceph-mon[38106]: pgmap v1577: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:39 smithi164 ceph-mon[33554]: pgmap v1578: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:39 smithi136 ceph-mon[38106]: pgmap v1578: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:39 smithi136 ceph-mon[29560]: pgmap v1578: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:51:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:41 smithi164 ceph-mon[33554]: pgmap v1579: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:41 smithi136 ceph-mon[38106]: pgmap v1579: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:41 smithi136 ceph-mon[29560]: pgmap v1579: 196 pgs: 196 active+clean; 582 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:43.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:51:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:51:44.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:43 smithi164 ceph-mon[33554]: pgmap v1580: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:44.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:43 smithi136 ceph-mon[38106]: pgmap v1580: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:43 smithi136 ceph-mon[29560]: pgmap v1580: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:45.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:45.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:45 smithi164 ceph-mon[33554]: pgmap v1581: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:45 smithi136 ceph-mon[38106]: pgmap v1581: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:46.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:45 smithi136 ceph-mon[29560]: pgmap v1581: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:51:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:51:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:51:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:47 smithi164 ceph-mon[33554]: pgmap v1582: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:48.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:47 smithi136 ceph-mon[29560]: pgmap v1582: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:48.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:47 smithi136 ceph-mon[38106]: pgmap v1582: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:49 smithi164 ceph-mon[33554]: pgmap v1583: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:49 smithi136 ceph-mon[38106]: pgmap v1583: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:49 smithi136 ceph-mon[29560]: pgmap v1583: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:51 smithi164 ceph-mon[33554]: pgmap v1584: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:51 smithi136 ceph-mon[38106]: pgmap v1584: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:51 smithi136 ceph-mon[29560]: pgmap v1584: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:53.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:51:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:51:54.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:53 smithi164 ceph-mon[33554]: pgmap v1585: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:53 smithi136 ceph-mon[38106]: pgmap v1585: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:53 smithi136 ceph-mon[29560]: pgmap v1585: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T05:51:55.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:55.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:55.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:51:56.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:55 smithi164 ceph-mon[33554]: pgmap v1586: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:55 smithi136 ceph-mon[29560]: pgmap v1586: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:55 smithi136 ceph-mon[38106]: pgmap v1586: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:51:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:51:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:51:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:51:58.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:57 smithi164 ceph-mon[33554]: pgmap v1587: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:57 smithi136 ceph-mon[29560]: pgmap v1587: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:57 smithi136 ceph-mon[38106]: pgmap v1587: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:51:59.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:51:59.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:00.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:51:59 smithi164 ceph-mon[33554]: pgmap v1588: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:00.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:51:59 smithi136 ceph-mon[38106]: pgmap v1588: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:00.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:51:59 smithi136 ceph-mon[29560]: pgmap v1588: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:02.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:01 smithi164 ceph-mon[33554]: pgmap v1589: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:02.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:01 smithi136 ceph-mon[29560]: pgmap v1589: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:01 smithi136 ceph-mon[38106]: pgmap v1589: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:03.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:52:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:52:04.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:03 smithi164 ceph-mon[33554]: pgmap v1590: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:03 smithi136 ceph-mon[29560]: pgmap v1590: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:03 smithi136 ceph-mon[38106]: pgmap v1590: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:05 smithi136 ceph-mon[38106]: pgmap v1591: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:05 smithi136 ceph-mon[29560]: pgmap v1591: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:05 smithi164 ceph-mon[33554]: pgmap v1591: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:52:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:52:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:52:08.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:07 smithi136 ceph-mon[38106]: pgmap v1592: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:08.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:07 smithi136 ceph-mon[29560]: pgmap v1592: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:08.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:07 smithi164 ceph-mon[33554]: pgmap v1592: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:10.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:09 smithi136 ceph-mon[38106]: pgmap v1593: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:10.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:09 smithi136 ceph-mon[29560]: pgmap v1593: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:09 smithi164 ceph-mon[33554]: pgmap v1593: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:11 smithi136 ceph-mon[38106]: pgmap v1594: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:11 smithi136 ceph-mon[29560]: pgmap v1594: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:11 smithi164 ceph-mon[33554]: pgmap v1594: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:13.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:52:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:52:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:13 smithi136 ceph-mon[38106]: pgmap v1595: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:13 smithi136 ceph-mon[29560]: pgmap v1595: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:13 smithi164 ceph-mon[33554]: pgmap v1595: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:15.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:15.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:16.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:15 smithi136 ceph-mon[38106]: pgmap v1596: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:16.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:15 smithi136 ceph-mon[29560]: pgmap v1596: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:16.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:15 smithi164 ceph-mon[33554]: pgmap v1596: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:52:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:52:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:52:18.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:18 smithi136 ceph-mon[38106]: pgmap v1597: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:18.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:18 smithi136 ceph-mon[29560]: pgmap v1597: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:18.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:18 smithi164 ceph-mon[33554]: pgmap v1597: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:20.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:20 smithi136 ceph-mon[38106]: pgmap v1598: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:20.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:20 smithi136 ceph-mon[29560]: pgmap v1598: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:20.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:20 smithi164 ceph-mon[33554]: pgmap v1598: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:22.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:22 smithi136 ceph-mon[38106]: pgmap v1599: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:22.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:22 smithi136 ceph-mon[29560]: pgmap v1599: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:22.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:22 smithi164 ceph-mon[33554]: pgmap v1599: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T05:52:23.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:52:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:52:24.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:24 smithi136 ceph-mon[38106]: pgmap v1600: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:24.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:24 smithi136 ceph-mon[29560]: pgmap v1600: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:24.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:24 smithi164 ceph-mon[33554]: pgmap v1600: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:25.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:25 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:25.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:25 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:25.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:25 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:26 smithi136 ceph-mon[29560]: pgmap v1601: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:26 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:52:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:26 smithi136 ceph-mon[38106]: pgmap v1601: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:26 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:52:26.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:26 smithi164 ceph-mon[33554]: pgmap v1601: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:26.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:26 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:52:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:52:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:52:27.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:27 smithi164 ceph-mon[33554]: pgmap v1602: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:52:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:52:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[38106]: pgmap v1602: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:52:27.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:27 smithi136 ceph-mon[29560]: pgmap v1602: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:29 smithi136 ceph-mon[38106]: pgmap v1603: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:29.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:29 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:29 smithi136 ceph-mon[29560]: pgmap v1603: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:29.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:29 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:29 smithi164 ceph-mon[33554]: pgmap v1603: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:29 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:31.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:31 smithi136 ceph-mon[38106]: pgmap v1604: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:31.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:31 smithi136 ceph-mon[29560]: pgmap v1604: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:31 smithi164 ceph-mon[33554]: pgmap v1604: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:33.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:33 smithi164 ceph-mon[33554]: pgmap v1605: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:33.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:52:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:52:33.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:33 smithi136 ceph-mon[38106]: pgmap v1605: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:33.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:33 smithi136 ceph-mon[29560]: pgmap v1605: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:34.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:34.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:35.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:35 smithi136 ceph-mon[38106]: pgmap v1606: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:35.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:35 smithi136 ceph-mon[29560]: pgmap v1606: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:35.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:35 smithi164 ceph-mon[33554]: pgmap v1606: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:37.496 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:52:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:52:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:52:37.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:37 smithi136 ceph-mon[38106]: pgmap v1607: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:37.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:37 smithi136 ceph-mon[29560]: pgmap v1607: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:37 smithi164 ceph-mon[33554]: pgmap v1607: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:39.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:39 smithi136 ceph-mon[38106]: pgmap v1608: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:39.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:39 smithi136 ceph-mon[29560]: pgmap v1608: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:39 smithi164 ceph-mon[33554]: pgmap v1608: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:41.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:41 smithi136 ceph-mon[38106]: pgmap v1609: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:41.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:41 smithi136 ceph-mon[29560]: pgmap v1609: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:41 smithi164 ceph-mon[33554]: pgmap v1609: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:43.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:43 smithi164 ceph-mon[33554]: pgmap v1610: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:43.772 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:52:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:52:43.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:43 smithi136 ceph-mon[38106]: pgmap v1610: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:43.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:43 smithi136 ceph-mon[29560]: pgmap v1610: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:44.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:44.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:44 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:44.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:44.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:44 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:44 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:45.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:45 smithi136 ceph-mon[38106]: pgmap v1611: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:45.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:45 smithi136 ceph-mon[29560]: pgmap v1611: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:45 smithi164 ceph-mon[33554]: pgmap v1611: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:47.514 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:52:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:52:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:52:47.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:47 smithi136 ceph-mon[38106]: pgmap v1612: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:47.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:47 smithi136 ceph-mon[29560]: pgmap v1612: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:47 smithi164 ceph-mon[33554]: pgmap v1612: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:49.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:49 smithi136 ceph-mon[38106]: pgmap v1613: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:49.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:49 smithi136 ceph-mon[29560]: pgmap v1613: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:49 smithi164 ceph-mon[33554]: pgmap v1613: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:51 smithi136 ceph-mon[38106]: pgmap v1614: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:51.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:51 smithi136 ceph-mon[29560]: pgmap v1614: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:51 smithi164 ceph-mon[33554]: pgmap v1614: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:52:53.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:53 smithi136 ceph-mon[38106]: pgmap v1615: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:53.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:53 smithi136 ceph-mon[29560]: pgmap v1615: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:53 smithi164 ceph-mon[33554]: pgmap v1615: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:53.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:52:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:52:54.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:54.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:52:55.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:55 smithi136 ceph-mon[38106]: pgmap v1616: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:55.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:55 smithi136 ceph-mon[29560]: pgmap v1616: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:55 smithi164 ceph-mon[33554]: pgmap v1616: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:57.537 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:52:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:52:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:52:57.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:57 smithi136 ceph-mon[38106]: pgmap v1617: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:57.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:57 smithi136 ceph-mon[29560]: pgmap v1617: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:57 smithi164 ceph-mon[33554]: pgmap v1617: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:52:58.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:58.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:52:59.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:52:59 smithi136 ceph-mon[38106]: pgmap v1618: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:59.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:52:59 smithi136 ceph-mon[29560]: pgmap v1618: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:52:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:52:59 smithi164 ceph-mon[33554]: pgmap v1618: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:53:01.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:01 smithi136 ceph-mon[38106]: pgmap v1619: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:01.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:01 smithi136 ceph-mon[29560]: pgmap v1619: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:01 smithi164 ceph-mon[33554]: pgmap v1619: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:03.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:03 smithi164 ceph-mon[33554]: pgmap v1620: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:03.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:53:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:53:03.720 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:53:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a[63607]: logger=cleanup t=2024-03-26T05:53:03.352809726Z level=info msg="Completed cleanup jobs" duration=2.587831ms 2024-03-26T05:53:03.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:03 smithi136 ceph-mon[38106]: pgmap v1620: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:03.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:03 smithi136 ceph-mon[29560]: pgmap v1620: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:04.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:04.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:05.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:05 smithi136 ceph-mon[38106]: pgmap v1621: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:05.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:05 smithi136 ceph-mon[29560]: pgmap v1621: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:05 smithi164 ceph-mon[33554]: pgmap v1621: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:07.562 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:53:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:53:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:53:07.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:07 smithi136 ceph-mon[38106]: pgmap v1622: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:07.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:07 smithi136 ceph-mon[29560]: pgmap v1622: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:07 smithi164 ceph-mon[33554]: pgmap v1622: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:09.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:09 smithi136 ceph-mon[38106]: pgmap v1623: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:53:09.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:09 smithi136 ceph-mon[29560]: pgmap v1623: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:53:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:09 smithi164 ceph-mon[33554]: pgmap v1623: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:53:11.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:11 smithi136 ceph-mon[38106]: pgmap v1624: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:11.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:11 smithi136 ceph-mon[29560]: pgmap v1624: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:11 smithi164 ceph-mon[33554]: pgmap v1624: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:13.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:13 smithi164 ceph-mon[33554]: pgmap v1625: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:13.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:13.772 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:53:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:53:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:13 smithi136 ceph-mon[38106]: pgmap v1625: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:13.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:13 smithi136 ceph-mon[29560]: pgmap v1625: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:13.825 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:14.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:15.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:15.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:15 smithi164 ceph-mon[33554]: pgmap v1626: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:15.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:15 smithi136 ceph-mon[38106]: pgmap v1626: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:15.976 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:15 smithi136 ceph-mon[29560]: pgmap v1626: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:53:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:53:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:53:17.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:17 smithi164 ceph-mon[33554]: pgmap v1627: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:17 smithi136 ceph-mon[38106]: pgmap v1627: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:17 smithi136 ceph-mon[29560]: pgmap v1627: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:19.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:19 smithi164 ceph-mon[33554]: pgmap v1628: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:19 smithi136 ceph-mon[38106]: pgmap v1628: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:20.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:19 smithi136 ceph-mon[29560]: pgmap v1628: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:21.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:21 smithi164 ceph-mon[33554]: pgmap v1629: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:21 smithi136 ceph-mon[29560]: pgmap v1629: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:21 smithi136 ceph-mon[38106]: pgmap v1629: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:23 smithi164 ceph-mon[33554]: pgmap v1630: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:23.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:53:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:53:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:23 smithi136 ceph-mon[29560]: pgmap v1630: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:23 smithi136 ceph-mon[38106]: pgmap v1630: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:24.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:25.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:25 smithi164 ceph-mon[33554]: pgmap v1631: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:25 smithi136 ceph-mon[38106]: pgmap v1631: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:25 smithi136 ceph-mon[29560]: pgmap v1631: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:26.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:26 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:53:27.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:26 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:53:27.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:26 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:53:27.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:53:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:53:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:27 smithi164 ceph-mon[33554]: pgmap v1632: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:53:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:53:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:53:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-mon[38106]: pgmap v1632: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:53:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:53:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:53:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-mon[29560]: pgmap v1632: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:53:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:53:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:53:28.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:29.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:29.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:29 smithi164 ceph-mon[33554]: pgmap v1633: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:29 smithi136 ceph-mon[38106]: pgmap v1633: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:29 smithi136 ceph-mon[29560]: pgmap v1633: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:31 smithi164 ceph-mon[33554]: pgmap v1634: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:31 smithi136 ceph-mon[29560]: pgmap v1634: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:31 smithi136 ceph-mon[38106]: pgmap v1634: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:33 smithi164 ceph-mon[33554]: pgmap v1635: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:33.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:53:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:53:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:33 smithi136 ceph-mon[29560]: pgmap v1635: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:33 smithi136 ceph-mon[38106]: pgmap v1635: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:34.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:35.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:35.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:35.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:35 smithi164 ceph-mon[33554]: pgmap v1636: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:36.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:35 smithi136 ceph-mon[38106]: pgmap v1636: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:36.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:35 smithi136 ceph-mon[29560]: pgmap v1636: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:53:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:53:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:53:37.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:37 smithi164 ceph-mon[33554]: pgmap v1637: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:38.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:37 smithi136 ceph-mon[38106]: pgmap v1637: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:38.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:37 smithi136 ceph-mon[29560]: pgmap v1637: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:39.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:39 smithi164 ceph-mon[33554]: pgmap v1638: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:40.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:39 smithi136 ceph-mon[29560]: pgmap v1638: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:40.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:39 smithi136 ceph-mon[38106]: pgmap v1638: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:41.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:41 smithi164 ceph-mon[33554]: pgmap v1639: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:42.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:41 smithi136 ceph-mon[38106]: pgmap v1639: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:42.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:41 smithi136 ceph-mon[29560]: pgmap v1639: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:43 smithi164 ceph-mon[33554]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:43.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:43.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:53:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:53:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:43 smithi136 ceph-mon[38106]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:44.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:43 smithi136 ceph-mon[29560]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:44.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:44.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:45.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:45 smithi164 ceph-mon[33554]: pgmap v1641: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:53:45.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:45 smithi136 ceph-mon[38106]: pgmap v1641: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:53:45.976 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:45 smithi136 ceph-mon[29560]: pgmap v1641: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:53:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:53:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:53:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:53:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:47 smithi164 ceph-mon[33554]: pgmap v1642: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:47 smithi136 ceph-mon[38106]: pgmap v1642: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:47 smithi136 ceph-mon[29560]: pgmap v1642: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:49 smithi164 ceph-mon[33554]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:49 smithi136 ceph-mon[38106]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:50.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:49 smithi136 ceph-mon[29560]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:51 smithi164 ceph-mon[33554]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:51 smithi136 ceph-mon[38106]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:51 smithi136 ceph-mon[29560]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:53 smithi164 ceph-mon[33554]: pgmap v1645: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:53.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:53:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:53:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:53 smithi136 ceph-mon[38106]: pgmap v1645: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:54.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:53 smithi136 ceph-mon[29560]: pgmap v1645: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:53:55.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:55 smithi164 ceph-mon[33554]: pgmap v1646: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:55 smithi136 ceph-mon[29560]: pgmap v1646: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:55 smithi136 ceph-mon[38106]: pgmap v1646: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:53:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:53:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:53:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:53:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:57 smithi164 ceph-mon[33554]: pgmap v1647: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:57 smithi136 ceph-mon[38106]: pgmap v1647: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:57 smithi136 ceph-mon[29560]: pgmap v1647: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:53:58.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:58 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:59.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:58 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:59.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:58 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:53:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:53:59 smithi164 ceph-mon[33554]: pgmap v1648: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:53:59 smithi136 ceph-mon[38106]: pgmap v1648: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:00.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:53:59 smithi136 ceph-mon[29560]: pgmap v1648: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:01 smithi164 ceph-mon[33554]: pgmap v1649: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:02.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:01 smithi136 ceph-mon[29560]: pgmap v1649: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:01 smithi136 ceph-mon[38106]: pgmap v1649: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:03 smithi164 ceph-mon[33554]: pgmap v1650: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:03.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:54:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:54:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:03 smithi136 ceph-mon[29560]: pgmap v1650: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:03 smithi136 ceph-mon[38106]: pgmap v1650: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:05.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:05.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:05 smithi164 ceph-mon[33554]: pgmap v1651: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:05 smithi136 ceph-mon[38106]: pgmap v1651: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:06.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:05 smithi136 ceph-mon[29560]: pgmap v1651: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:54:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:54:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:54:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:07 smithi164 ceph-mon[33554]: pgmap v1652: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:08.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:07 smithi136 ceph-mon[29560]: pgmap v1652: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:07 smithi136 ceph-mon[38106]: pgmap v1652: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:09 smithi164 ceph-mon[33554]: pgmap v1653: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:54:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:09 smithi136 ceph-mon[38106]: pgmap v1653: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:54:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:09 smithi136 ceph-mon[29560]: pgmap v1653: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:54:11.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:11 smithi164 ceph-mon[33554]: pgmap v1654: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:12.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:11 smithi136 ceph-mon[38106]: pgmap v1654: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:11 smithi136 ceph-mon[29560]: pgmap v1654: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:13 smithi164 ceph-mon[33554]: pgmap v1655: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:13.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:13.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:54:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:54:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:13 smithi136 ceph-mon[38106]: pgmap v1655: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:13 smithi136 ceph-mon[29560]: pgmap v1655: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:14.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:15.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:15.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:15 smithi136 ceph-mon[38106]: pgmap v1656: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:15.976 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:15 smithi136 ceph-mon[29560]: pgmap v1656: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:16.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:15 smithi164 ceph-mon[33554]: pgmap v1656: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:54:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:54:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:54:18.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:17 smithi136 ceph-mon[38106]: pgmap v1657: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:18.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:17 smithi136 ceph-mon[29560]: pgmap v1657: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:18.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:17 smithi164 ceph-mon[33554]: pgmap v1657: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:20.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:19 smithi136 ceph-mon[38106]: pgmap v1658: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:20.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:19 smithi136 ceph-mon[29560]: pgmap v1658: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:20.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:19 smithi164 ceph-mon[33554]: pgmap v1658: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:21 smithi136 ceph-mon[38106]: pgmap v1659: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:22.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:21 smithi136 ceph-mon[38106]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T05:54:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:21 smithi136 ceph-mon[29560]: pgmap v1659: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:22.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:21 smithi136 ceph-mon[29560]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T05:54:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:21 smithi164 ceph-mon[33554]: pgmap v1659: 196 pgs: 196 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:54:22.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:21 smithi164 ceph-mon[33554]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T05:54:23.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:22 smithi136 ceph-mon[38106]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T05:54:23.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:22 smithi136 ceph-mon[29560]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T05:54:23.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:22 smithi164 ceph-mon[33554]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T05:54:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:23 smithi164 ceph-mon[33554]: pgmap v1662: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:23.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:23 smithi164 ceph-mon[33554]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T05:54:23.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:54:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:54:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:23 smithi136 ceph-mon[29560]: pgmap v1662: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:24.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:23 smithi136 ceph-mon[29560]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T05:54:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:23 smithi136 ceph-mon[38106]: pgmap v1662: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:24.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:23 smithi136 ceph-mon[38106]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T05:54:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:25.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:24 smithi136 ceph-mon[38106]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T05:54:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:25.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:24 smithi136 ceph-mon[29560]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T05:54:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:25.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:24 smithi164 ceph-mon[33554]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T05:54:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:25 smithi136 ceph-mon[38106]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:25 smithi136 ceph-mon[38106]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:26.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:25 smithi136 ceph-mon[38106]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T05:54:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:25 smithi136 ceph-mon[29560]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:25 smithi136 ceph-mon[29560]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:26.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:25 smithi136 ceph-mon[29560]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T05:54:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:25 smithi164 ceph-mon[33554]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:25 smithi164 ceph-mon[33554]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:26.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:25 smithi164 ceph-mon[33554]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T05:54:27.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:26 smithi136 ceph-mon[38106]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T05:54:27.078 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:26 smithi136 ceph-mon[29560]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T05:54:27.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:26 smithi164 ceph-mon[33554]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T05:54:27.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:54:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:54:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[38106]: pgmap v1668: 196 pgs: 196 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:54:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[38106]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T05:54:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:54:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:54:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:54:28.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:54:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[29560]: pgmap v1668: 196 pgs: 196 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:54:28.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[29560]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T05:54:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:54:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:54:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:54:28.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:27 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:54:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:27 smithi164 ceph-mon[33554]: pgmap v1668: 196 pgs: 196 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:54:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:27 smithi164 ceph-mon[33554]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T05:54:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:54:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:54:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:27 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:54:28.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:27 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:54:29.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:28 smithi136 ceph-mon[38106]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T05:54:29.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:29.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:28 smithi136 ceph-mon[29560]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T05:54:29.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:28 smithi164 ceph-mon[33554]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T05:54:29.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:29 smithi136 ceph-mon[38106]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:30.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:29 smithi136 ceph-mon[38106]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T05:54:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:29 smithi136 ceph-mon[29560]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:30.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:29 smithi136 ceph-mon[29560]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T05:54:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:29 smithi164 ceph-mon[33554]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:30.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:29 smithi164 ceph-mon[33554]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T05:54:31.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:30 smithi136 ceph-mon[38106]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T05:54:31.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:30 smithi136 ceph-mon[29560]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T05:54:31.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:30 smithi164 ceph-mon[33554]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T05:54:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:31 smithi136 ceph-mon[38106]: pgmap v1674: 164 pgs: 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:31 smithi136 ceph-mon[38106]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:31 smithi136 ceph-mon[38106]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T05:54:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:31 smithi136 ceph-mon[29560]: pgmap v1674: 164 pgs: 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:31 smithi136 ceph-mon[29560]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:31 smithi136 ceph-mon[29560]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T05:54:32.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:31 smithi164 ceph-mon[33554]: pgmap v1674: 164 pgs: 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:32.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:31 smithi164 ceph-mon[33554]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:32.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:31 smithi164 ceph-mon[33554]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T05:54:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:32 smithi136 ceph-mon[38106]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T05:54:33.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:32 smithi136 ceph-mon[29560]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T05:54:33.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:32 smithi164 ceph-mon[33554]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T05:54:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:33 smithi164 ceph-mon[33554]: pgmap v1677: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:33 smithi164 ceph-mon[33554]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T05:54:33.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:54:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:54:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:33 smithi136 ceph-mon[38106]: pgmap v1677: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:34.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:33 smithi136 ceph-mon[38106]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T05:54:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:33 smithi136 ceph-mon[29560]: pgmap v1677: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:33 smithi136 ceph-mon[29560]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T05:54:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:34 smithi164 ceph-mon[33554]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T05:54:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:34 smithi136 ceph-mon[38106]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T05:54:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:34 smithi136 ceph-mon[29560]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T05:54:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:35 smithi164 ceph-mon[33554]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:35 smithi164 ceph-mon[33554]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T05:54:36.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:35 smithi136 ceph-mon[38106]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:35 smithi136 ceph-mon[38106]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T05:54:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:35 smithi136 ceph-mon[29560]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:35 smithi136 ceph-mon[29560]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T05:54:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:36 smithi164 ceph-mon[33554]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T05:54:37.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:36 smithi136 ceph-mon[38106]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T05:54:37.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:36 smithi136 ceph-mon[29560]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T05:54:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:54:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:54:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:54:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:37 smithi164 ceph-mon[33554]: pgmap v1683: 164 pgs: 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:37 smithi164 ceph-mon[33554]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:37 smithi164 ceph-mon[33554]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T05:54:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:37 smithi136 ceph-mon[29560]: pgmap v1683: 164 pgs: 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:37 smithi136 ceph-mon[29560]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:37 smithi136 ceph-mon[29560]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T05:54:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:37 smithi136 ceph-mon[38106]: pgmap v1683: 164 pgs: 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:37 smithi136 ceph-mon[38106]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:37 smithi136 ceph-mon[38106]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T05:54:39.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:38 smithi164 ceph-mon[33554]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T05:54:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:38 smithi136 ceph-mon[38106]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T05:54:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:38 smithi136 ceph-mon[29560]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T05:54:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:39 smithi164 ceph-mon[33554]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:39 smithi164 ceph-mon[33554]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T05:54:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:39 smithi136 ceph-mon[38106]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:40.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:39 smithi136 ceph-mon[38106]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T05:54:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:39 smithi136 ceph-mon[29560]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:39 smithi136 ceph-mon[29560]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T05:54:41.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:40 smithi164 ceph-mon[33554]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T05:54:41.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:40 smithi136 ceph-mon[29560]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T05:54:41.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:40 smithi136 ceph-mon[38106]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T05:54:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:41 smithi164 ceph-mon[33554]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:42.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:41 smithi164 ceph-mon[33554]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T05:54:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:41 smithi136 ceph-mon[29560]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:42.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:41 smithi136 ceph-mon[29560]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T05:54:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:41 smithi136 ceph-mon[38106]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 817 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:42.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:41 smithi136 ceph-mon[38106]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T05:54:43.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:42 smithi164 ceph-mon[33554]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T05:54:43.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:42 smithi136 ceph-mon[29560]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T05:54:43.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:42 smithi136 ceph-mon[38106]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T05:54:43.901 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:54:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:54:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:43 smithi164 ceph-mon[33554]: pgmap v1692: 164 pgs: 164 active+clean; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:43 smithi164 ceph-mon[33554]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:43 smithi164 ceph-mon[33554]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T05:54:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:43 smithi136 ceph-mon[29560]: pgmap v1692: 164 pgs: 164 active+clean; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:43 smithi136 ceph-mon[29560]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:43 smithi136 ceph-mon[29560]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T05:54:44.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:43 smithi136 ceph-mon[38106]: pgmap v1692: 164 pgs: 164 active+clean; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:43 smithi136 ceph-mon[38106]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:43 smithi136 ceph-mon[38106]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T05:54:44.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:45.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:44 smithi164 ceph-mon[33554]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T05:54:45.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:45.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:44 smithi136 ceph-mon[38106]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T05:54:45.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:45.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:44 smithi136 ceph-mon[29560]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T05:54:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:45 smithi164 ceph-mon[33554]: pgmap v1695: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:46.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:45 smithi164 ceph-mon[33554]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T05:54:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:45 smithi136 ceph-mon[38106]: pgmap v1695: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:46.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:45 smithi136 ceph-mon[38106]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T05:54:46.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:45 smithi136 ceph-mon[29560]: pgmap v1695: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 818 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:46.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:45 smithi136 ceph-mon[29560]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T05:54:47.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:46 smithi164 ceph-mon[33554]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T05:54:47.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:46 smithi136 ceph-mon[38106]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T05:54:47.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:46 smithi136 ceph-mon[29560]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T05:54:47.574 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:54:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:54:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:54:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:47 smithi164 ceph-mon[33554]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:48.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:47 smithi164 ceph-mon[33554]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T05:54:48.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:47 smithi136 ceph-mon[38106]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:48.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:47 smithi136 ceph-mon[38106]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T05:54:48.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:47 smithi136 ceph-mon[29560]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:48.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:47 smithi136 ceph-mon[29560]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T05:54:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:48 smithi164 ceph-mon[33554]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T05:54:49.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:48 smithi136 ceph-mon[38106]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T05:54:49.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:48 smithi136 ceph-mon[29560]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T05:54:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:49 smithi164 ceph-mon[33554]: pgmap v1701: 164 pgs: 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:49 smithi164 ceph-mon[33554]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:50.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:49 smithi164 ceph-mon[33554]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T05:54:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:49 smithi136 ceph-mon[38106]: pgmap v1701: 164 pgs: 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:49 smithi136 ceph-mon[38106]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:50.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:49 smithi136 ceph-mon[38106]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T05:54:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:49 smithi136 ceph-mon[29560]: pgmap v1701: 164 pgs: 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:49 smithi136 ceph-mon[29560]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:50.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:49 smithi136 ceph-mon[29560]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T05:54:51.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:50 smithi164 ceph-mon[33554]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T05:54:51.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:50 smithi136 ceph-mon[38106]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T05:54:51.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:50 smithi136 ceph-mon[29560]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T05:54:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:51 smithi164 ceph-mon[33554]: pgmap v1704: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:52.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:51 smithi164 ceph-mon[33554]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T05:54:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:51 smithi136 ceph-mon[38106]: pgmap v1704: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:52.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:51 smithi136 ceph-mon[38106]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T05:54:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:51 smithi136 ceph-mon[29560]: pgmap v1704: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:54:52.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:51 smithi136 ceph-mon[29560]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T05:54:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:52 smithi136 ceph-mon[38106]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T05:54:53.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:52 smithi136 ceph-mon[38106]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T05:54:53.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:52 smithi136 ceph-mon[29560]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T05:54:53.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:52 smithi136 ceph-mon[29560]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T05:54:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:52 smithi164 ceph-mon[33554]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T05:54:53.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:52 smithi164 ceph-mon[33554]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T05:54:53.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:54:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:54:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:53 smithi136 ceph-mon[38106]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:54.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:53 smithi136 ceph-mon[38106]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T05:54:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:53 smithi136 ceph-mon[29560]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:54.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:53 smithi136 ceph-mon[29560]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T05:54:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:53 smithi164 ceph-mon[33554]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:54.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:53 smithi164 ceph-mon[33554]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T05:54:55.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:55.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:54 smithi136 ceph-mon[38106]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:55.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:55.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:54 smithi136 ceph-mon[29560]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:54:55.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:54 smithi164 ceph-mon[33554]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:54:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:56 smithi136 ceph-mon[38106]: pgmap v1710: 164 pgs: 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:56.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:56 smithi136 ceph-mon[38106]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T05:54:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:56 smithi136 ceph-mon[29560]: pgmap v1710: 164 pgs: 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:56.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:56 smithi136 ceph-mon[29560]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T05:54:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:56 smithi164 ceph-mon[33554]: pgmap v1710: 164 pgs: 164 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:56.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:56 smithi164 ceph-mon[33554]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T05:54:57.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:57 smithi136 ceph-mon[38106]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T05:54:57.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:57 smithi136 ceph-mon[29560]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T05:54:57.325 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:54:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:54:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:54:57.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:57 smithi164 ceph-mon[33554]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T05:54:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:58 smithi136 ceph-mon[38106]: pgmap v1713: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:58.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:58 smithi136 ceph-mon[38106]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T05:54:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:58 smithi136 ceph-mon[29560]: pgmap v1713: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:58.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:58 smithi136 ceph-mon[29560]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T05:54:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:58 smithi164 ceph-mon[33554]: pgmap v1713: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:54:58.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:58 smithi164 ceph-mon[33554]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T05:54:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:59 smithi136 ceph-mon[38106]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T05:54:59.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:54:59 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:59 smithi136 ceph-mon[29560]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T05:54:59.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:54:59 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:54:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:59 smithi164 ceph-mon[33554]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T05:54:59.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:54:59 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:00.027 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: Running main() from gmock_main.cc 2024-03-26T05:55:00.027 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-26T05:55:00.027 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [----------] Global test environment set-up. 2024-03-26T05:55:00.027 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-26T05:55:00.027 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (1808564 ms) 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3025 ms) 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3016 ms) 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3025 ms) 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3021 ms) 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3017 ms) 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3020 ms) 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3014 ms) 2024-03-26T05:55:00.028 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3035 ms) 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3020 ms) 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3021 ms) 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3017 ms) 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3029 ms) 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3019 ms) 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (1847845 ms total) 2024-03-26T05:55:00.029 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: 2024-03-26T05:55:00.030 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [----------] Global test environment tear-down 2024-03-26T05:55:00.030 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (1847845 ms total) 2024-03-26T05:55:00.030 INFO:tasks.workunit.client.0.smithi136.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-26T05:55:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75052 2024-03-26T05:55:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75052 2024-03-26T05:55:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75560 2024-03-26T05:55:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75560 2024-03-26T05:55:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75904 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75904 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74957 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74957 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75435 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75435 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75747 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75747 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75514 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75514 2024-03-26T05:55:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75778 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75778 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75274 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75274 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74876 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74876 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75875 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75875 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75156 2024-03-26T05:55:00.034 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75156 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74693 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74693 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74752 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74752 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75111 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75111 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=74792 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 74792 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.035 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75195 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75195 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75211 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75211 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75696 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75696 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=75932 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 75932 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ exit 0 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ cleanup 2024-03-26T05:55:00.036 INFO:tasks.workunit.client.0.smithi136.stderr:+ pkill -P 74687 2024-03-26T05:55:00.045 INFO:tasks.workunit.client.0.smithi136.stderr:+ true 2024-03-26T05:55:00.046 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T05:55:00.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T05:55:00.078 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-26T05:55:00.079 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-26T05:55:00.079 DEBUG:teuthology.orchestra.run.smithi136:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-26T05:55:00.138 INFO:tasks.workunit.client.0.smithi136.stderr:++ uuidgen 2024-03-26T05:55:00.146 INFO:tasks.workunit.client.0.smithi136.stderr:+ p=78bcab27-41f1-4625-8adb-19d91c406d36 2024-03-26T05:55:00.146 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create 78bcab27-41f1-4625-8adb-19d91c406d36 12 2024-03-26T05:55:00.231 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.237+0000 7fcfe1393640 1 -- 172.21.15.136:0/2154748762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100e50 msgr2=0x7fcfdc10d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.237+0000 7fcfe1393640 1 --2- 172.21.15.136:0/2154748762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100e50 0x7fcfdc10d310 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009890 tx=0x7fcfcc02bb70 comp rx=0 tx=0).stop 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 -- 172.21.15.136:0/2154748762 shutdown_connections 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 --2- 172.21.15.136:0/2154748762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100e50 0x7fcfdc10d310 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 --2- 172.21.15.136:0/2154748762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfdc100490 0x7fcfdc100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 --2- 172.21.15.136:0/2154748762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfdc106550 0x7fcfdc106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 -- 172.21.15.136:0/2154748762 >> 172.21.15.136:0/2154748762 conn(0x7fcfdc077670 msgr2=0x7fcfdc077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 -- 172.21.15.136:0/2154748762 shutdown_connections 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 -- 172.21.15.136:0/2154748762 wait complete. 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 Processor -- start 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.238+0000 7fcfe1393640 1 -- start start 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfe1393640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100490 0x7fcfdc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfe1393640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfdc100e50 0x7fcfdc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfe1393640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfdc106550 0x7fcfdc1a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfe1393640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc198e00 con 0x7fcfdc106550 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfe1393640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc1a3620 con 0x7fcfdc100e50 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfe1393640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc1a3760 con 0x7fcfdc100490 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100490 0x7fcfdc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfdc106550 0x7fcfdc1a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100490 0x7fcfdc198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:42862/0 (socket says 172.21.15.136:42862) 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 -- 172.21.15.136:0/485930745 learned_addr learned my addr 172.21.15.136:0/485930745 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfda7fc640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfdc100e50 0x7fcfdc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 -- 172.21.15.136:0/485930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfdc100e50 msgr2=0x7fcfdc1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfdc100e50 0x7fcfdc1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 -- 172.21.15.136:0/485930745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfdc106550 msgr2=0x7fcfdc1a30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfdc106550 0x7fcfdc1a30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 -- 172.21.15.136:0/485930745 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4009560 con 0x7fcfdc100490 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdb7fe640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfdc106550 0x7fcfdc1a30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:55:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfda7fc640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfdc100e50 0x7fcfdc1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T05:55:00.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfdaffd640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100490 0x7fcfdc198060 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9c0 tx=0x7fcfd400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:00.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfc3fff640 1 -- 172.21.15.136:0/485930745 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400cde0 con 0x7fcfdc100490 2024-03-26T05:55:00.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfc3fff640 1 -- 172.21.15.136:0/485930745 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fcfd40044d0 con 0x7fcfdc100490 2024-03-26T05:55:00.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc009490 con 0x7fcfdc100490 2024-03-26T05:55:00.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfc3fff640 1 -- 172.21.15.136:0/485930745 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4010430 con 0x7fcfdc100490 2024-03-26T05:55:00.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.239+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfdc1a3d90 con 0x7fcfdc100490 2024-03-26T05:55:00.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.240+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfdc0688c0 con 0x7fcfdc100490 2024-03-26T05:55:00.235 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.241+0000 7fcfc3fff640 1 -- 172.21.15.136:0/485930745 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7fcfd40035f0 con 0x7fcfdc100490 2024-03-26T05:55:00.235 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.241+0000 7fcfc3fff640 1 --2- 172.21.15.136:0/485930745 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfc406d9f0 0x7fcfc406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:00.235 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.241+0000 7fcfc3fff640 1 -- 172.21.15.136:0/485930745 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(803..803 src has 263..803) v4 ==== 7374+0+0 (secure 0 0 0) 0x7fcfd4014070 con 0x7fcfdc100490 2024-03-26T05:55:00.235 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.241+0000 7fcfda7fc640 1 --2- 172.21.15.136:0/485930745 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfc406d9f0 0x7fcfc406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:00.235 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.241+0000 7fcfda7fc640 1 --2- 172.21.15.136:0/485930745 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfc406d9f0 0x7fcfc406feb0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc06a800 tx=0x7fcfd00093e0 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:00.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.246+0000 7fcfc3fff640 1 -- 172.21.15.136:0/485930745 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcfd405ac10 con 0x7fcfdc100490 2024-03-26T05:55:00.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:00 smithi136 ceph-mon[38106]: pgmap v1716: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:00.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:00 smithi136 ceph-mon[38106]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T05:55:00.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:00 smithi136 ceph-mon[29560]: pgmap v1716: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:00.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:00 smithi136 ceph-mon[29560]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T05:55:00.411 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:00.420+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12} v 0) v1 -- 0x7fcfdc10fca0 con 0x7fcfdc100490 2024-03-26T05:55:00.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:00 smithi164 ceph-mon[33554]: pgmap v1716: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:00.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:00 smithi164 ceph-mon[33554]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T05:55:01.031 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.039+0000 7fcfc3fff640 1 -- 172.21.15.136:0/485930745 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]=0 pool '78bcab27-41f1-4625-8adb-19d91c406d36' created v804) v1 ==== 176+0+0 (secure 0 0 0) 0x7fcfd405a7a0 con 0x7fcfdc100490 2024-03-26T05:55:01.139 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.147+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12} v 0) v1 -- 0x7fcfdc06a2a0 con 0x7fcfdc100490 2024-03-26T05:55:01.139 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.148+0000 7fcfc3fff640 1 -- 172.21.15.136:0/485930745 <== mon.2 v2:172.21.15.136:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]=0 pool '78bcab27-41f1-4625-8adb-19d91c406d36' already exists v804) v1 ==== 183+0+0 (secure 0 0 0) 0x7fcfd405a7a0 con 0x7fcfdc100490 2024-03-26T05:55:01.139 INFO:tasks.workunit.client.0.smithi136.stderr:pool '78bcab27-41f1-4625-8adb-19d91c406d36' already exists 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfc406d9f0 msgr2=0x7fcfc406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 --2- 172.21.15.136:0/485930745 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfc406d9f0 0x7fcfc406feb0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc06a800 tx=0x7fcfd00093e0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100490 msgr2=0x7fcfdc198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100490 0x7fcfdc198060 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9c0 tx=0x7fcfd400ee80 comp rx=0 tx=0).stop 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 shutdown_connections 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 --2- 172.21.15.136:0/485930745 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcfc406d9f0 0x7fcfc406feb0 unknown :-1 s=CLOSED pgs=3530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfdc106550 0x7fcfdc1a30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfdc100e50 0x7fcfdc1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.142 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 --2- 172.21.15.136:0/485930745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfdc100490 0x7fcfdc198060 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.143 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 >> 172.21.15.136:0/485930745 conn(0x7fcfdc077670 msgr2=0x7fcfdc104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:01.143 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 shutdown_connections 2024-03-26T05:55:01.143 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.150+0000 7fcfe1393640 1 -- 172.21.15.136:0/485930745 wait complete. 2024-03-26T05:55:01.151 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set-quota 78bcab27-41f1-4625-8adb-19d91c406d36 max_objects 10 2024-03-26T05:55:01.236 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 -- 172.21.15.136:0/4044787166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100e50 msgr2=0x7f3d8010d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 --2- 172.21.15.136:0/4044787166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100e50 0x7f3d8010d310 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009cd0 tx=0x7f3d7802c0f0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 -- 172.21.15.136:0/4044787166 shutdown_connections 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 --2- 172.21.15.136:0/4044787166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100e50 0x7f3d8010d310 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 --2- 172.21.15.136:0/4044787166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d80100490 0x7f3d80100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 --2- 172.21.15.136:0/4044787166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d80106550 0x7f3d80106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 -- 172.21.15.136:0/4044787166 >> 172.21.15.136:0/4044787166 conn(0x7f3d80077580 msgr2=0x7f3d80077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 -- 172.21.15.136:0/4044787166 shutdown_connections 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.242+0000 7f3d8853a640 1 -- 172.21.15.136:0/4044787166 wait complete. 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d8853a640 1 Processor -- start 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d8853a640 1 -- start start 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d8853a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100490 0x7f3d801980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d8853a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d80100e50 0x7f3d80198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:01.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d8853a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d80106550 0x7f3d801a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d8853a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d80198e60 con 0x7f3d80106550 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d8853a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d801a3680 con 0x7f3d80100e50 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d8853a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d801a37c0 con 0x7f3d80100490 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d862d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100490 0x7f3d801980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d86ad7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d80106550 0x7f3d801a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d862d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100490 0x7f3d801980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:36976/0 (socket says 172.21.15.136:36976) 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d862d6640 1 -- 172.21.15.136:0/3761780185 learned_addr learned my addr 172.21.15.136:0/3761780185 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d85ad5640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d80100e50 0x7f3d80198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d86ad7640 1 -- 172.21.15.136:0/3761780185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100490 msgr2=0x7f3d801980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d86ad7640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100490 0x7f3d801980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d86ad7640 1 -- 172.21.15.136:0/3761780185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d80100e50 msgr2=0x7f3d80198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d86ad7640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d80100e50 0x7f3d80198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:01.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d86ad7640 1 -- 172.21.15.136:0/3761780185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c009490 con 0x7f3d80106550 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.243+0000 7f3d862d6640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100490 0x7f3d801980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.244+0000 7f3d86ad7640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d80106550 0x7f3d801a3140 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f3d7802c630 tx=0x7f3d78032a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.244+0000 7f3d777fe640 1 -- 172.21.15.136:0/3761780185 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78034c70 con 0x7f3d80106550 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.244+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d78009900 con 0x7f3d80106550 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.244+0000 7f3d777fe640 1 -- 172.21.15.136:0/3761780185 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3d78003e20 con 0x7f3d80106550 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.244+0000 7f3d777fe640 1 -- 172.21.15.136:0/3761780185 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d780047c0 con 0x7f3d80106550 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.244+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d801a3d00 con 0x7f3d80106550 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.245+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d80101bb0 con 0x7f3d80106550 2024-03-26T05:55:01.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.245+0000 7f3d777fe640 1 -- 172.21.15.136:0/3761780185 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f3d7803b040 con 0x7f3d80106550 2024-03-26T05:55:01.240 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.246+0000 7f3d777fe640 1 --2- 172.21.15.136:0/3761780185 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3d6c06d9f0 0x7f3d6c06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:01.240 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.246+0000 7f3d862d6640 1 --2- 172.21.15.136:0/3761780185 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3d6c06d9f0 0x7f3d6c06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:01.240 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.246+0000 7f3d777fe640 1 -- 172.21.15.136:0/3761780185 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(804..804 src has 263..804) v4 ==== 7749+0+0 (secure 0 0 0) 0x7f3d780afec0 con 0x7f3d80106550 2024-03-26T05:55:01.240 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.246+0000 7f3d862d6640 1 --2- 172.21.15.136:0/3761780185 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3d6c06d9f0 0x7f3d6c06feb0 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c011690 tx=0x7f3d7c005f90 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:01.242 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.251+0000 7f3d777fe640 1 -- 172.21.15.136:0/3761780185 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3d7807cdc0 con 0x7f3d80106550 2024-03-26T05:55:01.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:01 smithi136 ceph-mon[38106]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T05:55:01.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:01 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/485930745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:01.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:01 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:01.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:01 smithi136 ceph-mon[29560]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T05:55:01.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:01 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/485930745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:01.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:01 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:01.418 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:01.426+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f3d8006a2e0 con 0x7f3d80106550 2024-03-26T05:55:01.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:01 smithi164 ceph-mon[33554]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T05:55:01.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:01 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/485930745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:01.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:01 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:02.028 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:02.036+0000 7f3d777fe640 1 -- 172.21.15.136:0/3761780185 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v805) v1 ==== 223+0+0 (secure 0 0 0) 0x7f3d7807c950 con 0x7f3d80106550 2024-03-26T05:55:02.141 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:02.150+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f3d8010fec0 con 0x7f3d80106550 2024-03-26T05:55:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[38106]: pgmap v1719: 164 pgs: 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:55:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[38106]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:55:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]': finished 2024-03-26T05:55:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[38106]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T05:55:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/485930745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:02.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:55:02.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[29560]: pgmap v1719: 164 pgs: 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:55:02.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[29560]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:55:02.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]': finished 2024-03-26T05:55:02.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[29560]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T05:55:02.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/485930745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:02.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:02.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:02 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:55:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:02 smithi164 ceph-mon[33554]: pgmap v1719: 164 pgs: 164 active+clean; 582 KiB data, 823 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:55:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:02 smithi164 ceph-mon[33554]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:55:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]': finished 2024-03-26T05:55:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:02 smithi164 ceph-mon[33554]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T05:55:02.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/485930745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:02.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:02 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pg_num": 12}]: dispatch 2024-03-26T05:55:02.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:02 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:55:03.043 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.051+0000 7f3d777fe640 1 -- 172.21.15.136:0/3761780185 <== mon.0 v2:172.21.15.136:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v806) v1 ==== 223+0+0 (secure 0 0 0) 0x7f3d780410d0 con 0x7f3d80106550 2024-03-26T05:55:03.043 INFO:tasks.workunit.client.0.smithi136.stderr:set-quota max_objects = 10 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3d6c06d9f0 msgr2=0x7f3d6c06feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 --2- 172.21.15.136:0/3761780185 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3d6c06d9f0 0x7f3d6c06feb0 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c011690 tx=0x7f3d7c005f90 comp rx=0 tx=0).stop 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d80106550 msgr2=0x7f3d801a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d80106550 0x7f3d801a3140 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f3d7802c630 tx=0x7f3d78032a20 comp rx=0 tx=0).stop 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 shutdown_connections 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 --2- 172.21.15.136:0/3761780185 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3d6c06d9f0 0x7f3d6c06feb0 unknown :-1 s=CLOSED pgs=3531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d80106550 0x7f3d801a3140 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d80100e50 0x7f3d80198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 --2- 172.21.15.136:0/3761780185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d80100490 0x7f3d801980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.048 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 >> 172.21.15.136:0/3761780185 conn(0x7f3d80077580 msgr2=0x7f3d80104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:03.048 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 shutdown_connections 2024-03-26T05:55:03.048 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.055+0000 7f3d8853a640 1 -- 172.21.15.136:0/3761780185 wait complete. 2024-03-26T05:55:03.058 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool application enable 78bcab27-41f1-4625-8adb-19d91c406d36 rados 2024-03-26T05:55:03.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 -- 172.21.15.136:0/1219095819 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a101027b0 msgr2=0x7f2a10102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:03.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 --2- 172.21.15.136:0/1219095819 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a101027b0 0x7f2a10102b90 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009860 tx=0x7f2a0002bc00 comp rx=0 tx=0).stop 2024-03-26T05:55:03.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 -- 172.21.15.136:0/1219095819 shutdown_connections 2024-03-26T05:55:03.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 --2- 172.21.15.136:0/1219095819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a1010bb80 0x7f2a1010dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 --2- 172.21.15.136:0/1219095819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a101030d0 0x7f2a1010b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 --2- 172.21.15.136:0/1219095819 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a101027b0 0x7f2a10102b90 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 -- 172.21.15.136:0/1219095819 >> 172.21.15.136:0/1219095819 conn(0x7f2a10077610 msgr2=0x7f2a10077a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 -- 172.21.15.136:0/1219095819 shutdown_connections 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.151+0000 7f2a16a15640 1 -- 172.21.15.136:0/1219095819 wait complete. 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a16a15640 1 Processor -- start 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a16a15640 1 -- start start 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a16a15640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a101027b0 0x7f2a10197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a16a15640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a101030d0 0x7f2a101978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a16a15640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1010bb80 0x7f2a1019bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a16a15640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1019c1e0 con 0x7f2a101027b0 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a16a15640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1019c350 con 0x7f2a101030d0 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a16a15640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1019c510 con 0x7f2a1010bb80 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a0ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a101027b0 0x7f2a10197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:03.146 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a14fb2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1010bb80 0x7f2a1019bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a0ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a101027b0 0x7f2a10197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:54000/0 (socket says 172.21.15.136:54000) 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a14fb2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1010bb80 0x7f2a1019bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:36998/0 (socket says 172.21.15.136:36998) 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a0ffff640 1 -- 172.21.15.136:0/3358280890 learned_addr learned my addr 172.21.15.136:0/3358280890 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a0f7fe640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a101030d0 0x7f2a101978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a0ffff640 1 -- 172.21.15.136:0/3358280890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1010bb80 msgr2=0x7f2a1019bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a0ffff640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1010bb80 0x7f2a1019bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a0ffff640 1 -- 172.21.15.136:0/3358280890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a101030d0 msgr2=0x7f2a101978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.152+0000 7f2a0ffff640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a101030d0 0x7f2a101978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a0ffff640 1 -- 172.21.15.136:0/3358280890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08009900 con 0x7f2a101027b0 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a0f7fe640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a101030d0 0x7f2a101978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a14fb2640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1010bb80 0x7f2a1019bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a0ffff640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a101027b0 0x7f2a10197370 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f2a00037040 tx=0x7f2a000346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a0d7fa640 1 -- 172.21.15.136:0/3358280890 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0002c7a0 con 0x7f2a101027b0 2024-03-26T05:55:03.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a00009490 con 0x7f2a101027b0 2024-03-26T05:55:03.148 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a0d7fa640 1 -- 172.21.15.136:0/3358280890 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f2a000035f0 con 0x7f2a101027b0 2024-03-26T05:55:03.148 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a0d7fa640 1 -- 172.21.15.136:0/3358280890 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0003d650 con 0x7f2a101027b0 2024-03-26T05:55:03.148 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.153+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a101a3e90 con 0x7f2a101027b0 2024-03-26T05:55:03.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.154+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a100688c0 con 0x7f2a101027b0 2024-03-26T05:55:03.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.154+0000 7f2a0d7fa640 1 -- 172.21.15.136:0/3358280890 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f2a0003b040 con 0x7f2a101027b0 2024-03-26T05:55:03.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.155+0000 7f2a0d7fa640 1 --2- 172.21.15.136:0/3358280890 >> v2:172.21.15.136:6800/3380174081 conn(0x7f29fc06d9f0 0x7f29fc06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:03.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.155+0000 7f2a0f7fe640 1 --2- 172.21.15.136:0/3358280890 >> v2:172.21.15.136:6800/3380174081 conn(0x7f29fc06d9f0 0x7f29fc06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:03.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.155+0000 7f2a0d7fa640 1 -- 172.21.15.136:0/3358280890 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(806..806 src has 263..806) v4 ==== 7749+0+0 (secure 0 0 0) 0x7f2a000af4e0 con 0x7f2a101027b0 2024-03-26T05:55:03.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.155+0000 7f2a0f7fe640 1 --2- 172.21.15.136:0/3358280890 >> v2:172.21.15.136:6800/3380174081 conn(0x7f29fc06d9f0 0x7f29fc06feb0 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f2a10077dc0 tx=0x7f29f8005e50 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:03.151 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.160+0000 7f2a0d7fa640 1 -- 172.21.15.136:0/3358280890 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2a0007c3e0 con 0x7f2a101027b0 2024-03-26T05:55:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:55:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:03 smithi136 ceph-mon[38106]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T05:55:03.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:03 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:55:03.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:55:03.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:03 smithi136 ceph-mon[29560]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T05:55:03.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:03 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:55:03.328 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:03.337+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"} v 0) v1 -- 0x7f2a10106380 con 0x7f2a101027b0 2024-03-26T05:55:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:55:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:03 smithi164 ceph-mon[33554]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T05:55:03.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:03 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:55:03.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:55:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:55:04.049 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:04.057+0000 7f2a0d7fa640 1 -- 172.21.15.136:0/3358280890 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]=0 enabled application 'rados' on pool '78bcab27-41f1-4625-8adb-19d91c406d36' v807) v1 ==== 213+0+0 (secure 0 0 0) 0x7f2a0007bf70 con 0x7f2a101027b0 2024-03-26T05:55:04.162 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:04.171+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"} v 0) v1 -- 0x7f2a10110f00 con 0x7f2a101027b0 2024-03-26T05:55:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:04 smithi136 ceph-mon[38106]: pgmap v1722: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:55:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:04 smithi136 ceph-mon[38106]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T05:55:04.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:04 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]: dispatch 2024-03-26T05:55:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:04 smithi136 ceph-mon[29560]: pgmap v1722: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:55:04.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:04 smithi136 ceph-mon[29560]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T05:55:04.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:04 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]: dispatch 2024-03-26T05:55:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:04 smithi164 ceph-mon[33554]: pgmap v1722: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:04.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3761780185' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:55:04.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:04 smithi164 ceph-mon[33554]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T05:55:04.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:04 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]: dispatch 2024-03-26T05:55:05.056 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.065+0000 7f2a0d7fa640 1 -- 172.21.15.136:0/3358280890 <== mon.0 v2:172.21.15.136:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]=0 enabled application 'rados' on pool '78bcab27-41f1-4625-8adb-19d91c406d36' v808) v1 ==== 213+0+0 (secure 0 0 0) 0x7f2a000410d0 con 0x7f2a101027b0 2024-03-26T05:55:05.056 INFO:tasks.workunit.client.0.smithi136.stderr:enabled application 'rados' on pool '78bcab27-41f1-4625-8adb-19d91c406d36' 2024-03-26T05:55:05.061 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 >> v2:172.21.15.136:6800/3380174081 conn(0x7f29fc06d9f0 msgr2=0x7f29fc06feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:05.061 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 --2- 172.21.15.136:0/3358280890 >> v2:172.21.15.136:6800/3380174081 conn(0x7f29fc06d9f0 0x7f29fc06feb0 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f2a10077dc0 tx=0x7f29f8005e50 comp rx=0 tx=0).stop 2024-03-26T05:55:05.061 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a101027b0 msgr2=0x7f2a10197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:05.061 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a101027b0 0x7f2a10197370 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f2a00037040 tx=0x7f2a000346f0 comp rx=0 tx=0).stop 2024-03-26T05:55:05.061 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 shutdown_connections 2024-03-26T05:55:05.061 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 --2- 172.21.15.136:0/3358280890 >> v2:172.21.15.136:6800/3380174081 conn(0x7f29fc06d9f0 0x7f29fc06feb0 unknown :-1 s=CLOSED pgs=3532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:05.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1010bb80 0x7f2a1019bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:05.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a101030d0 0x7f2a101978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:05.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 --2- 172.21.15.136:0/3358280890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a101027b0 0x7f2a10197370 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:05.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 >> 172.21.15.136:0/3358280890 conn(0x7f2a10077610 msgr2=0x7f2a10103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:05.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 shutdown_connections 2024-03-26T05:55:05.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:05.069+0000 7f2a16a15640 1 -- 172.21.15.136:0/3358280890 wait complete. 2024-03-26T05:55:05.073 INFO:tasks.workunit.client.0.smithi136.stderr:++ seq 1 10 2024-03-26T05:55:05.083 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.083 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj1 /etc/passwd 2024-03-26T05:55:05.123 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.124 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj2 /etc/passwd 2024-03-26T05:55:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj3 /etc/passwd 2024-03-26T05:55:05.195 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.195 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj4 /etc/passwd 2024-03-26T05:55:05.230 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.230 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj5 /etc/passwd 2024-03-26T05:55:05.264 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.264 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj6 /etc/passwd 2024-03-26T05:55:05.300 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.300 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj7 /etc/passwd 2024-03-26T05:55:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:05 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]': finished 2024-03-26T05:55:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:05 smithi136 ceph-mon[38106]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T05:55:05.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:05 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]: dispatch 2024-03-26T05:55:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:05 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]': finished 2024-03-26T05:55:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:05 smithi136 ceph-mon[29560]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T05:55:05.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:05 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]: dispatch 2024-03-26T05:55:05.335 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.335 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj8 /etc/passwd 2024-03-26T05:55:05.370 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.370 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj9 /etc/passwd 2024-03-26T05:55:05.404 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:55:05.404 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put obj10 /etc/passwd 2024-03-26T05:55:05.438 INFO:tasks.workunit.client.0.smithi136.stderr:+ sleep 30 2024-03-26T05:55:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:05 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]': finished 2024-03-26T05:55:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:05 smithi164 ceph-mon[33554]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T05:55:05.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:05 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]: dispatch 2024-03-26T05:55:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:06 smithi136 ceph-mon[38106]: pgmap v1725: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:06 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]': finished 2024-03-26T05:55:06.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:06 smithi136 ceph-mon[38106]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T05:55:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:06 smithi136 ceph-mon[29560]: pgmap v1725: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:06 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]': finished 2024-03-26T05:55:06.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:06 smithi136 ceph-mon[29560]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T05:55:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:06 smithi164 ceph-mon[33554]: pgmap v1725: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:06 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3358280890' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "app": "rados"}]': finished 2024-03-26T05:55:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:06 smithi164 ceph-mon[33554]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T05:55:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:55:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:55:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:55:08.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:08 smithi136 ceph-mon[38106]: pgmap v1727: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-03-26T05:55:08.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:08 smithi136 ceph-mon[29560]: pgmap v1727: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-03-26T05:55:08.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:08 smithi164 ceph-mon[33554]: pgmap v1727: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-03-26T05:55:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:10 smithi136 ceph-mon[38106]: pgmap v1728: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-26T05:55:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:10 smithi136 ceph-mon[38106]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' is full (reached quota's max_objects: 10) 2024-03-26T05:55:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:10 smithi136 ceph-mon[38106]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:55:10.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:10 smithi136 ceph-mon[38106]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T05:55:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:10 smithi136 ceph-mon[29560]: pgmap v1728: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-26T05:55:10.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:10 smithi136 ceph-mon[29560]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' is full (reached quota's max_objects: 10) 2024-03-26T05:55:10.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:10 smithi136 ceph-mon[29560]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:55:10.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:10 smithi136 ceph-mon[29560]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T05:55:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:10 smithi164 ceph-mon[33554]: pgmap v1728: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-26T05:55:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:10 smithi164 ceph-mon[33554]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' is full (reached quota's max_objects: 10) 2024-03-26T05:55:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:10 smithi164 ceph-mon[33554]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:55:10.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:10 smithi164 ceph-mon[33554]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T05:55:12.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:12 smithi136 ceph-mon[38106]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 796 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-26T05:55:12.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:12 smithi136 ceph-mon[29560]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 796 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-26T05:55:12.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:12 smithi164 ceph-mon[33554]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 796 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-26T05:55:13.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:55:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:55:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:14 smithi136 ceph-mon[38106]: pgmap v1731: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T05:55:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:14 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:14.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:14 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:14 smithi136 ceph-mon[29560]: pgmap v1731: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T05:55:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:14 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:14.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:14 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:14 smithi164 ceph-mon[33554]: pgmap v1731: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T05:55:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:14 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:14.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:14 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:15.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:15 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:15.508 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:15 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:15.508 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:15 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:16.444 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:16 smithi136 ceph-mon[38106]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-26T05:55:16.445 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:16 smithi136 ceph-mon[29560]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-26T05:55:16.468 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:16 smithi164 ceph-mon[33554]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-26T05:55:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:55:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:55:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:55:18.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:18 smithi164 ceph-mon[33554]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:18.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:18 smithi136 ceph-mon[38106]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:18.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:18 smithi136 ceph-mon[29560]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:20.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:20 smithi164 ceph-mon[33554]: pgmap v1734: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:20 smithi136 ceph-mon[29560]: pgmap v1734: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:20.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:20 smithi136 ceph-mon[38106]: pgmap v1734: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:22.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:22 smithi164 ceph-mon[33554]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-26T05:55:22.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:22 smithi136 ceph-mon[29560]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-26T05:55:22.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:22 smithi136 ceph-mon[38106]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-26T05:55:23.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:55:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:55:24.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:24 smithi164 ceph-mon[33554]: pgmap v1736: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:24.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:24 smithi136 ceph-mon[29560]: pgmap v1736: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:24 smithi136 ceph-mon[38106]: pgmap v1736: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:25.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:25 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:25.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:25 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:25.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:25 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:26.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:26 smithi136 ceph-mon[38106]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:26.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:26 smithi136 ceph-mon[29560]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:26.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:26 smithi164 ceph-mon[33554]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:27.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:27 smithi164 ceph-mon[33554]: pgmap v1738: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:27.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:27 smithi136 ceph-mon[38106]: pgmap v1738: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:27.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:27 smithi136 ceph-mon[29560]: pgmap v1738: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:27.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:55:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:55:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:55:28.430 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:55:28.431 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:55:28.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:55:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:29 smithi164 ceph-mon[33554]: pgmap v1739: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:29.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:29 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:29.481 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:29 smithi136 ceph-mon[38106]: pgmap v1739: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:29.481 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:29 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:29.481 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:29 smithi136 ceph-mon[29560]: pgmap v1739: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:29.481 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:29 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:31.594 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:31 smithi164 ceph-mon[33554]: pgmap v1740: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:31.594 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:31 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:31.594 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:31 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:31 smithi136 ceph-mon[38106]: pgmap v1740: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:31 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:32.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:31 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:31 smithi136 ceph-mon[29560]: pgmap v1740: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:31 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:32.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:31 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:32 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:32 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:32 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:55:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:32 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:55:33.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:32 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:55:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:55:33.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:55:33.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:55:33.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:32 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:33.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:33 smithi164 ceph-mon[33554]: pgmap v1741: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:33.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:55:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:55:34.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:33 smithi136 ceph-mon[38106]: pgmap v1741: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:34.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:33 smithi136 ceph-mon[29560]: pgmap v1741: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:35.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:35.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:35.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:35.439 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=151765 2024-03-26T05:55:35.439 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set-quota 78bcab27-41f1-4625-8adb-19d91c406d36 max_objects 100 2024-03-26T05:55:35.439 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put onemore /etc/passwd 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 -- 172.21.15.136:0/2998197609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648101f40 msgr2=0x7f264810e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2998197609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648101f40 0x7f264810e400 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f2638009ca0 tx=0x7f263802c0f0 comp rx=0 tx=0).stop 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 -- 172.21.15.136:0/2998197609 shutdown_connections 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2998197609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648101f40 0x7f264810e400 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2998197609 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2648101580 0x7f2648101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2998197609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2648107640 0x7f2648107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 -- 172.21.15.136:0/2998197609 >> 172.21.15.136:0/2998197609 conn(0x7f26480fd070 msgr2=0x7f26480ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 -- 172.21.15.136:0/2998197609 shutdown_connections 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 -- 172.21.15.136:0/2998197609 wait complete. 2024-03-26T05:55:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 Processor -- start 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.543+0000 7f264ffe2640 1 -- start start 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264ffe2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2648101580 0x7f26481971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264ffe2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2648101f40 0x7f2648197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264ffe2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648107640 0x7f264819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264ffe2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264819c0a0 con 0x7f2648107640 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264ffe2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264819c210 con 0x7f2648101f40 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264ffe2640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264819c3d0 con 0x7f2648101580 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264dd7e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2648101580 0x7f26481971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264e57f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648107640 0x7f264819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264dd7e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2648101580 0x7f26481971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55140/0 (socket says 172.21.15.136:55140) 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264e57f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648107640 0x7f264819baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56662/0 (socket says 172.21.15.136:56662) 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264dd7e640 1 -- 172.21.15.136:0/2712568424 learned_addr learned my addr 172.21.15.136:0/2712568424 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264d57d640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2648101f40 0x7f2648197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:35.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264e57f640 1 -- 172.21.15.136:0/2712568424 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2648101580 msgr2=0x7f26481971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264e57f640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2648101580 0x7f26481971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264e57f640 1 -- 172.21.15.136:0/2712568424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2648101f40 msgr2=0x7f2648197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264e57f640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2648101f40 0x7f2648197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264e57f640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c009490 con 0x7f2648107640 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264dd7e640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2648101580 0x7f26481971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264e57f640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648107640 0x7f264819baa0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f263802c700 tx=0x7f2638034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638034c60 con 0x7f2648107640 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f2638032a40 con 0x7f2648107640 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2638009900 con 0x7f2648107640 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638004780 con 0x7f2648107640 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.544+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26481a3d50 con 0x7f2648107640 2024-03-26T05:55:35.539 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.546+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26480688c0 con 0x7f2648107640 2024-03-26T05:55:35.540 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.546+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f263803b040 con 0x7f2648107640 2024-03-26T05:55:35.540 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.546+0000 7f2646ffd640 1 --2- 172.21.15.136:0/2712568424 >> v2:172.21.15.136:6800/3380174081 conn(0x7f263006d9f0 0x7f263006feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:35.540 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.546+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(809..809 src has 263..809) v4 ==== 7762+0+0 (secure 0 0 0) 0x7f26380afe40 con 0x7f2648107640 2024-03-26T05:55:35.540 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.546+0000 7f264dd7e640 1 --2- 172.21.15.136:0/2712568424 >> v2:172.21.15.136:6800/3380174081 conn(0x7f263006d9f0 0x7f263006feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:35.541 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.546+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=810}) v3 -- 0x7f2630078ed0 con 0x7f2648107640 2024-03-26T05:55:35.541 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.547+0000 7f264dd7e640 1 --2- 172.21.15.136:0/2712568424 >> v2:172.21.15.136:6800/3380174081 conn(0x7f263006d9f0 0x7f263006feb0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7f26400058c0 tx=0x7f264000a380 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:35.542 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.551+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f263807cdb0 con 0x7f2648107640 2024-03-26T05:55:35.716 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.725+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f2648110d80 con 0x7f2648107640 2024-03-26T05:55:35.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.859+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v810) v1 ==== 225+0+0 (secure 0 0 0) 0x7f263807c940 con 0x7f2648107640 2024-03-26T05:55:35.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.859+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 8 ==== osd_map(810..810 src has 263..810) v4 ==== 628+0+0 (secure 0 0 0) 0x7f2638081080 con 0x7f2648107640 2024-03-26T05:55:35.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.859+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=811}) v3 -- 0x7f2630079df0 con 0x7f2648107640 2024-03-26T05:55:35.959 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:35.967+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f2648110f20 con 0x7f2648107640 2024-03-26T05:55:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:35 smithi164 ceph-mon[33554]: pgmap v1742: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:36.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:35 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:55:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:35 smithi136 ceph-mon[38106]: pgmap v1742: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:36.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:35 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:55:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:35 smithi136 ceph-mon[29560]: pgmap v1742: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:36.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:35 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:55:36.856 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.865+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v811) v1 ==== 225+0+0 (secure 0 0 0) 0x7f263804b070 con 0x7f2648107640 2024-03-26T05:55:36.857 INFO:tasks.workunit.client.0.smithi136.stderr:set-quota max_objects = 100 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 2024-03-26T05:55:36.857 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.866+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 <== mon.0 v2:172.21.15.136:3300/0 10 ==== osd_map(811..811 src has 263..811) v4 ==== 628+0+0 (secure 0 0 0) 0x7f2638085640 con 0x7f2648107640 2024-03-26T05:55:36.857 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.866+0000 7f2646ffd640 1 -- 172.21.15.136:0/2712568424 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=812}) v3 -- 0x7f2630079ee0 con 0x7f2648107640 2024-03-26T05:55:36.860 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.868+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 >> v2:172.21.15.136:6800/3380174081 conn(0x7f263006d9f0 msgr2=0x7f263006feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:36.860 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2712568424 >> v2:172.21.15.136:6800/3380174081 conn(0x7f263006d9f0 0x7f263006feb0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7f26400058c0 tx=0x7f264000a380 comp rx=0 tx=0).stop 2024-03-26T05:55:36.860 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648107640 msgr2=0x7f264819baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:36.860 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648107640 0x7f264819baa0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f263802c700 tx=0x7f2638034830 comp rx=0 tx=0).stop 2024-03-26T05:55:36.861 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 shutdown_connections 2024-03-26T05:55:36.861 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2712568424 >> v2:172.21.15.136:6800/3380174081 conn(0x7f263006d9f0 0x7f263006feb0 unknown :-1 s=CLOSED pgs=3544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:36.861 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2648107640 0x7f264819baa0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:36.861 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2648101f40 0x7f2648197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:36.861 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 --2- 172.21.15.136:0/2712568424 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2648101580 0x7f26481971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:36.861 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.869+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 >> 172.21.15.136:0/2712568424 conn(0x7f26480fd070 msgr2=0x7f26480fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:36.861 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.870+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 shutdown_connections 2024-03-26T05:55:36.862 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:36.870+0000 7f264ffe2640 1 -- 172.21.15.136:0/2712568424 wait complete. 2024-03-26T05:55:36.874 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 151765 2024-03-26T05:55:37.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:55:37.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:36 smithi164 ceph-mon[33554]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T05:55:37.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:36 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:55:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:55:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:36 smithi136 ceph-mon[38106]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T05:55:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:36 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:55:37.231 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:55:37.231 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:36 smithi136 ceph-mon[29560]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T05:55:37.231 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:36 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:55:37.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:55:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:55:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:55:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:37 smithi164 ceph-mon[33554]: pgmap v1744: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:38.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:37 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:55:38.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:37 smithi164 ceph-mon[33554]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T05:55:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:37 smithi136 ceph-mon[38106]: pgmap v1744: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:37 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:55:38.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:37 smithi136 ceph-mon[38106]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T05:55:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:37 smithi136 ceph-mon[29560]: pgmap v1744: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:38.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:37 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2712568424' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:55:38.326 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:37 smithi136 ceph-mon[29560]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T05:55:39.252 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T05:55:39.252 INFO:tasks.workunit.client.0.smithi136.stderr:+ true 2024-03-26T05:55:39.252 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put twomore /etc/passwd 2024-03-26T05:55:39.289 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set-quota 78bcab27-41f1-4625-8adb-19d91c406d36 max_bytes 100 2024-03-26T05:55:39.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.381+0000 7fb4d629a640 1 -- 172.21.15.136:0/3180531267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d0100450 msgr2=0x7fb4d01008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:39.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.381+0000 7fb4d629a640 1 --2- 172.21.15.136:0/3180531267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d0100450 0x7fb4d01008d0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009890 tx=0x7fb4c402bb70 comp rx=0 tx=0).stop 2024-03-26T05:55:39.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 -- 172.21.15.136:0/3180531267 shutdown_connections 2024-03-26T05:55:39.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 --2- 172.21.15.136:0/3180531267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4d0100e10 0x7fb4d010d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:39.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 --2- 172.21.15.136:0/3180531267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d0100450 0x7fb4d01008d0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:39.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 --2- 172.21.15.136:0/3180531267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4d01064f0 0x7fb4d01068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 -- 172.21.15.136:0/3180531267 >> 172.21.15.136:0/3180531267 conn(0x7fb4d00775b0 msgr2=0x7fb4d00779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 -- 172.21.15.136:0/3180531267 shutdown_connections 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 -- 172.21.15.136:0/3180531267 wait complete. 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 Processor -- start 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 -- start start 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4d0100450 0x7fb4d01961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4d0100e10 0x7fb4d0196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.382+0000 7fb4d629a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d01064f0 0x7fb4d019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4d629a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d019b090 con 0x7fb4d01064f0 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4d629a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d019b200 con 0x7fb4d0100e10 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4d629a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d019b3c0 con 0x7fb4d0100450 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4d0100450 0x7fb4d01961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4d0100450 0x7fb4d01961d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55196/0 (socket says 172.21.15.136:55196) 2024-03-26T05:55:39.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 -- 172.21.15.136:0/374602196 learned_addr learned my addr 172.21.15.136:0/374602196 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4d4837640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d01064f0 0x7fb4d019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cf7fe640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4d0100e10 0x7fb4d0196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 -- 172.21.15.136:0/374602196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4d0100e10 msgr2=0x7fb4d0196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4d0100e10 0x7fb4d0196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 -- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d01064f0 msgr2=0x7fb4d019aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d01064f0 0x7fb4d019aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 -- 172.21.15.136:0/374602196 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c8009900 con 0x7fb4d0100450 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cf7fe640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4d0100e10 0x7fb4d0196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4d4837640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d01064f0 0x7fb4d019aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cffff640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4d0100450 0x7fb4d01961d0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000d950 tx=0x7fb4c000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:39.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cd7fa640 1 -- 172.21.15.136:0/374602196 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0009c30 con 0x7fb4d0100450 2024-03-26T05:55:39.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cd7fa640 1 -- 172.21.15.136:0/374602196 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fb4c0009d90 con 0x7fb4d0100450 2024-03-26T05:55:39.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c4009490 con 0x7fb4d0100450 2024-03-26T05:55:39.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4cd7fa640 1 -- 172.21.15.136:0/374602196 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000f690 con 0x7fb4d0100450 2024-03-26T05:55:39.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.383+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d01a2d40 con 0x7fb4d0100450 2024-03-26T05:55:39.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.384+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d0068470 con 0x7fb4d0100450 2024-03-26T05:55:39.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.385+0000 7fb4cd7fa640 1 -- 172.21.15.136:0/374602196 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7fb4c0010460 con 0x7fb4d0100450 2024-03-26T05:55:39.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.385+0000 7fb4cd7fa640 1 --2- 172.21.15.136:0/374602196 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb4b806d9f0 0x7fb4b806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:55:39.380 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.385+0000 7fb4cd7fa640 1 -- 172.21.15.136:0/374602196 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(812..812 src has 263..812) v4 ==== 7762+0+0 (secure 0 0 0) 0x7fb4c008d5a0 con 0x7fb4d0100450 2024-03-26T05:55:39.380 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.385+0000 7fb4cf7fe640 1 --2- 172.21.15.136:0/374602196 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb4b806d9f0 0x7fb4b806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:55:39.380 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.385+0000 7fb4cf7fe640 1 --2- 172.21.15.136:0/374602196 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb4b806d9f0 0x7fb4b806feb0 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0069f70 tx=0x7fb4c4032340 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:55:39.381 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.390+0000 7fb4cd7fa640 1 -- 172.21.15.136:0/374602196 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb4c005a490 con 0x7fb4d0100450 2024-03-26T05:55:39.557 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:39.565+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7fb4d010fc30 con 0x7fb4d0100450 2024-03-26T05:55:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:39 smithi164 ceph-mon[33554]: pgmap v1746: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:39 smithi164 ceph-mon[33554]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:55:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:39 smithi164 ceph-mon[33554]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:55:40.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:39 smithi164 ceph-mon[33554]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T05:55:40.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:39 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/374602196' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:40.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:39 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:40.247 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:40.255+0000 7fb4cd7fa640 1 -- 172.21.15.136:0/374602196 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v813) v1 ==== 221+0+0 (secure 0 0 0) 0x7fb4c005a020 con 0x7fb4d0100450 2024-03-26T05:55:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[29560]: pgmap v1746: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[29560]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:55:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[29560]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:55:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[29560]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T05:55:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/374602196' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:40.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[38106]: pgmap v1746: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[38106]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:55:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[38106]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:55:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[38106]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T05:55:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/374602196' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:40.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:39 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:40.354 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:40.362+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7fb4d01a2ed0 con 0x7fb4d0100450 2024-03-26T05:55:41.252 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.261+0000 7fb4cd7fa640 1 -- 172.21.15.136:0/374602196 <== mon.2 v2:172.21.15.136:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v814) v1 ==== 221+0+0 (secure 0 0 0) 0x7fb4c0017070 con 0x7fb4d0100450 2024-03-26T05:55:41.253 INFO:tasks.workunit.client.0.smithi136.stderr:set-quota max_bytes = 100 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 2024-03-26T05:55:41.255 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.263+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb4b806d9f0 msgr2=0x7fb4b806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:41.255 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.263+0000 7fb4d629a640 1 --2- 172.21.15.136:0/374602196 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb4b806d9f0 0x7fb4b806feb0 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0069f70 tx=0x7fb4c4032340 comp rx=0 tx=0).stop 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.263+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4d0100450 msgr2=0x7fb4d01961d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.263+0000 7fb4d629a640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4d0100450 0x7fb4d01961d0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000d950 tx=0x7fb4c000de10 comp rx=0 tx=0).stop 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.264+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 shutdown_connections 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.264+0000 7fb4d629a640 1 --2- 172.21.15.136:0/374602196 >> v2:172.21.15.136:6800/3380174081 conn(0x7fb4b806d9f0 0x7fb4b806feb0 unknown :-1 s=CLOSED pgs=3546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.264+0000 7fb4d629a640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4d01064f0 0x7fb4d019aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.264+0000 7fb4d629a640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4d0100e10 0x7fb4d0196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.264+0000 7fb4d629a640 1 --2- 172.21.15.136:0/374602196 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4d0100450 0x7fb4d01961d0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.264+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 >> 172.21.15.136:0/374602196 conn(0x7fb4d00775b0 msgr2=0x7fb4d00ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.264+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 shutdown_connections 2024-03-26T05:55:41.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:55:41.264+0000 7fb4d629a640 1 -- 172.21.15.136:0/374602196 wait complete. 2024-03-26T05:55:41.266 INFO:tasks.workunit.client.0.smithi136.stderr:+ sleep 30 2024-03-26T05:55:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:55:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[38106]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T05:55:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/374602196' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[38106]: pgmap v1749: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:55:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:55:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[29560]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T05:55:41.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/374602196' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:41.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:41.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:41 smithi136 ceph-mon[29560]: pgmap v1749: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:55:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:55:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:41 smithi164 ceph-mon[33554]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T05:55:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:41 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/374602196' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:41 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:55:41.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:41 smithi164 ceph-mon[33554]: pgmap v1749: 176 pgs: 176 active+clean; 698 KiB data, 877 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:55:42.577 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:42 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:55:42.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:42 smithi136 ceph-mon[38106]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T05:55:42.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:42 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:55:42.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:42 smithi136 ceph-mon[29560]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T05:55:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:42 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:55:42.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:42 smithi164 ceph-mon[33554]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T05:55:43.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:43 smithi136 ceph-mon[38106]: pgmap v1751: 176 pgs: 176 active+clean; 721 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 909 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-03-26T05:55:43.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:43 smithi136 ceph-mon[29560]: pgmap v1751: 176 pgs: 176 active+clean; 721 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 909 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-03-26T05:55:43.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:43 smithi164 ceph-mon[33554]: pgmap v1751: 176 pgs: 176 active+clean; 721 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 909 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-03-26T05:55:43.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:55:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:55:44.887 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:44.888 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:44 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:44.888 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:44 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:44.888 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:44 smithi164 ceph-mon[33554]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' is full (reached quota's max_bytes: 100 B) 2024-03-26T05:55:44.888 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:44 smithi164 ceph-mon[33554]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:55:44.888 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:44 smithi164 ceph-mon[33554]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T05:55:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[29560]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' is full (reached quota's max_bytes: 100 B) 2024-03-26T05:55:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[29560]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:55:45.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[29560]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T05:55:45.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:45.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:45.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:55:45.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[38106]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' is full (reached quota's max_bytes: 100 B) 2024-03-26T05:55:45.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[38106]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:55:45.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:44 smithi136 ceph-mon[38106]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T05:55:45.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:45 smithi164 ceph-mon[33554]: pgmap v1753: 176 pgs: 176 active+clean; 721 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 976 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T05:55:46.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:45 smithi136 ceph-mon[29560]: pgmap v1753: 176 pgs: 176 active+clean; 721 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 976 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T05:55:46.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:45 smithi136 ceph-mon[38106]: pgmap v1753: 176 pgs: 176 active+clean; 721 KiB data, 877 MiB used, 714 GiB / 715 GiB avail; 976 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T05:55:47.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:55:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:55:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:55:47.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:47 smithi164 ceph-mon[33554]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-26T05:55:48.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:47 smithi136 ceph-mon[38106]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-26T05:55:48.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:47 smithi136 ceph-mon[29560]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-26T05:55:49.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:49 smithi164 ceph-mon[33554]: pgmap v1755: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:55:50.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:49 smithi136 ceph-mon[38106]: pgmap v1755: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:55:50.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:49 smithi136 ceph-mon[29560]: pgmap v1755: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:55:51.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:51 smithi164 ceph-mon[33554]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-26T05:55:52.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:51 smithi136 ceph-mon[38106]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-26T05:55:52.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:51 smithi136 ceph-mon[29560]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-26T05:55:53.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:53 smithi164 ceph-mon[33554]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:53.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:55:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:55:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:53 smithi136 ceph-mon[38106]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:53 smithi136 ceph-mon[29560]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:55:54.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:55.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:55.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:55:55.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:55 smithi164 ceph-mon[33554]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-26T05:55:56.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:55 smithi136 ceph-mon[38106]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-26T05:55:56.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:55 smithi136 ceph-mon[29560]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-26T05:55:57.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:55:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:55:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:55:57.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:57 smithi164 ceph-mon[33554]: pgmap v1759: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:58.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:57 smithi136 ceph-mon[38106]: pgmap v1759: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:58.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:57 smithi136 ceph-mon[29560]: pgmap v1759: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:55:59.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:59 smithi164 ceph-mon[33554]: pgmap v1760: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:55:59.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:59 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:55:59.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:55:59 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:59 smithi136 ceph-mon[38106]: pgmap v1760: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:59 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:56:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:55:59 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:00.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:59 smithi136 ceph-mon[29560]: pgmap v1760: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:00.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:59 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:56:00.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:55:59 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:01.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:01 smithi164 ceph-mon[33554]: pgmap v1761: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:02.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:01 smithi136 ceph-mon[38106]: pgmap v1761: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:01 smithi136 ceph-mon[29560]: pgmap v1761: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:03.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:03 smithi164 ceph-mon[33554]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:03.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:56:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:56:04.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:03 smithi136 ceph-mon[38106]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:04.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:03 smithi136 ceph-mon[29560]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:04.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:05.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:05.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:05.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:05 smithi164 ceph-mon[33554]: pgmap v1763: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:06.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:05 smithi136 ceph-mon[38106]: pgmap v1763: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:06.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:05 smithi136 ceph-mon[29560]: pgmap v1763: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:07.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:56:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:56:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:56:07.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:07 smithi164 ceph-mon[33554]: pgmap v1764: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:08.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:07 smithi136 ceph-mon[29560]: pgmap v1764: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:08.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:07 smithi136 ceph-mon[38106]: pgmap v1764: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:09.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:09 smithi164 ceph-mon[33554]: pgmap v1765: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:10.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:09 smithi136 ceph-mon[38106]: pgmap v1765: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:10.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:09 smithi136 ceph-mon[29560]: pgmap v1765: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:11.268 INFO:tasks.workunit.client.0.smithi136.stderr:+ pid=151866 2024-03-26T05:56:11.269 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set-quota 78bcab27-41f1-4625-8adb-19d91c406d36 max_bytes 0 2024-03-26T05:56:11.269 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put two /etc/passwd 2024-03-26T05:56:11.365 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 -- 172.21.15.136:0/431715809 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 msgr2=0x7f20bc102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:11.365 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 --2- 172.21.15.136:0/431715809 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 0x7f20bc102be0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009900 tx=0x7f20ac02bd60 comp rx=0 tx=0).stop 2024-03-26T05:56:11.365 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 -- 172.21.15.136:0/431715809 shutdown_connections 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 --2- 172.21.15.136:0/431715809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20bc10bb40 0x7f20bc10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 --2- 172.21.15.136:0/431715809 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20bc103120 0x7f20bc10b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 --2- 172.21.15.136:0/431715809 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 0x7f20bc102be0 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 -- 172.21.15.136:0/431715809 >> 172.21.15.136:0/431715809 conn(0x7f20bc0775b0 msgr2=0x7f20bc0779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 -- 172.21.15.136:0/431715809 shutdown_connections 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 -- 172.21.15.136:0/431715809 wait complete. 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 Processor -- start 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.371+0000 7f20c289a640 1 -- start start 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20c289a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 0x7f20bc199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20c289a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20bc103120 0x7f20bc199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:11.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20c289a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20bc10bb40 0x7f20bc19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20c289a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc19e3e0 con 0x7f20bc10bb40 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20c289a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc19e550 con 0x7f20bc103120 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20c289a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc19e710 con 0x7f20bc102800 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 0x7f20bc199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 0x7f20bc199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:52806/0 (socket says 172.21.15.136:52806) 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20c0e37640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20bc10bb40 0x7f20bc19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 -- 172.21.15.136:0/2433668571 learned_addr learned my addr 172.21.15.136:0/2433668571 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bb7fe640 1 --2- 172.21.15.136:0/2433668571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20bc103120 0x7f20bc199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 -- 172.21.15.136:0/2433668571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20bc103120 msgr2=0x7f20bc199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 --2- 172.21.15.136:0/2433668571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20bc103120 0x7f20bc199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 -- 172.21.15.136:0/2433668571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20bc10bb40 msgr2=0x7f20bc19dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 --2- 172.21.15.136:0/2433668571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20bc10bb40 0x7f20bc19dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b4009840 con 0x7f20bc102800 2024-03-26T05:56:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.372+0000 7f20bbfff640 1 --2- 172.21.15.136:0/2433668571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 0x7f20bc199520 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f20ac005940 tx=0x7f20ac02c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.373+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac0043d0 con 0x7f20bc102800 2024-03-26T05:56:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.373+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f20ac003aa0 con 0x7f20bc102800 2024-03-26T05:56:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.373+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac009560 con 0x7f20bc102800 2024-03-26T05:56:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.373+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac005270 con 0x7f20bc102800 2024-03-26T05:56:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.373+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20bc1a6090 con 0x7f20bc102800 2024-03-26T05:56:11.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.374+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20bc068470 con 0x7f20bc102800 2024-03-26T05:56:11.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.374+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f20ac0053d0 con 0x7f20bc102800 2024-03-26T05:56:11.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.374+0000 7f20b97fa640 1 --2- 172.21.15.136:0/2433668571 >> v2:172.21.15.136:6800/3380174081 conn(0x7f20a406d9f0 0x7f20a406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:11.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.375+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(815..815 src has 263..815) v4 ==== 7762+0+0 (secure 0 0 0) 0x7f20ac0b0190 con 0x7f20bc102800 2024-03-26T05:56:11.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.375+0000 7f20bb7fe640 1 --2- 172.21.15.136:0/2433668571 >> v2:172.21.15.136:6800/3380174081 conn(0x7f20a406d9f0 0x7f20a406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:11.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.375+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=816}) v3 -- 0x7f20a4078e50 con 0x7f20bc102800 2024-03-26T05:56:11.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.375+0000 7f20bb7fe640 1 --2- 172.21.15.136:0/2433668571 >> v2:172.21.15.136:6800/3380174081 conn(0x7f20a406d9f0 0x7f20a406feb0 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7f20bc077c90 tx=0x7f20b0005f90 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:11.371 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.379+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f20ac07cfd0 con 0x7f20bc102800 2024-03-26T05:56:11.546 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.553+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f20bc110d50 con 0x7f20bc102800 2024-03-26T05:56:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.728+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 7 ==== osd_map(816..816 src has 263..816) v4 ==== 628+0+0 (secure 0 0 0) 0x7f20ac003c10 con 0x7f20bc102800 2024-03-26T05:56:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.728+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=817}) v3 -- 0x7f20a4079980 con 0x7f20bc102800 2024-03-26T05:56:11.725 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.733+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v816) v1 ==== 217+0+0 (secure 0 0 0) 0x7f20ac043020 con 0x7f20bc102800 2024-03-26T05:56:11.833 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:11.841+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f20bc110f60 con 0x7f20bc102800 2024-03-26T05:56:12.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:11 smithi136 ceph-mon[29560]: pgmap v1766: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:11 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2433668571' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:12.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:11 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:12.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:11 smithi136 ceph-mon[38106]: pgmap v1766: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:12.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:11 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2433668571' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:12.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:11 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:12.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:11 smithi164 ceph-mon[33554]: pgmap v1766: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:12.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:11 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2433668571' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:12.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:11 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:12.736 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.745+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 9 ==== osd_map(817..817 src has 263..817) v4 ==== 628+0+0 (secure 0 0 0) 0x7f20ac03ce30 con 0x7f20bc102800 2024-03-26T05:56:12.736 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.745+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=818}) v3 -- 0x7f20a4079db0 con 0x7f20bc102800 2024-03-26T05:56:12.742 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.750+0000 7f20b97fa640 1 -- 172.21.15.136:0/2433668571 <== mon.2 v2:172.21.15.136:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v817) v1 ==== 217+0+0 (secure 0 0 0) 0x7f20ac043070 con 0x7f20bc102800 2024-03-26T05:56:12.742 INFO:tasks.workunit.client.0.smithi136.stderr:set-quota max_bytes = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 2024-03-26T05:56:12.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.753+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 >> v2:172.21.15.136:6800/3380174081 conn(0x7f20a406d9f0 msgr2=0x7f20a406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:12.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.753+0000 7f20c289a640 1 --2- 172.21.15.136:0/2433668571 >> v2:172.21.15.136:6800/3380174081 conn(0x7f20a406d9f0 0x7f20a406feb0 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7f20bc077c90 tx=0x7f20b0005f90 comp rx=0 tx=0).stop 2024-03-26T05:56:12.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 msgr2=0x7f20bc199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:12.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 --2- 172.21.15.136:0/2433668571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 0x7f20bc199520 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f20ac005940 tx=0x7f20ac02c8c0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 shutdown_connections 2024-03-26T05:56:12.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 --2- 172.21.15.136:0/2433668571 >> v2:172.21.15.136:6800/3380174081 conn(0x7f20a406d9f0 0x7f20a406feb0 unknown :-1 s=CLOSED pgs=3548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.747 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 --2- 172.21.15.136:0/2433668571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20bc10bb40 0x7f20bc19dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.747 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 --2- 172.21.15.136:0/2433668571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20bc103120 0x7f20bc199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.747 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 --2- 172.21.15.136:0/2433668571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20bc102800 0x7f20bc199520 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.747 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 >> 172.21.15.136:0/2433668571 conn(0x7f20bc0775b0 msgr2=0x7f20bc10a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:12.747 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 shutdown_connections 2024-03-26T05:56:12.747 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.754+0000 7f20c289a640 1 -- 172.21.15.136:0/2433668571 wait complete. 2024-03-26T05:56:12.757 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set-quota 78bcab27-41f1-4625-8adb-19d91c406d36 max_objects 0 2024-03-26T05:56:12.847 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 -- 172.21.15.136:0/2476462580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c0ffe50 msgr2=0x7f298c1002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:12.847 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 --2- 172.21.15.136:0/2476462580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c0ffe50 0x7f298c1002b0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f2980009930 tx=0x7f298002bc20 comp rx=0 tx=0).stop 2024-03-26T05:56:12.847 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 -- 172.21.15.136:0/2476462580 shutdown_connections 2024-03-26T05:56:12.847 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 --2- 172.21.15.136:0/2476462580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f298c1007f0 0x7f298c10cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 --2- 172.21.15.136:0/2476462580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c0ffe50 0x7f298c1002b0 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 --2- 172.21.15.136:0/2476462580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f298c104bc0 0x7f298c104fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 -- 172.21.15.136:0/2476462580 >> 172.21.15.136:0/2476462580 conn(0x7f298c0757b0 msgr2=0x7f298c075bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 -- 172.21.15.136:0/2476462580 shutdown_connections 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.852+0000 7f2991afb640 1 -- 172.21.15.136:0/2476462580 wait complete. 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f2991afb640 1 Processor -- start 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f2991afb640 1 -- start start 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f2991afb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f298c0ffe50 0x7f298c199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f2991afb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c1007f0 0x7f298c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f2991afb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f298c104bc0 0x7f298c19dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:12.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f2991afb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c19e2f0 con 0x7f298c104bc0 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f2991afb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c19e460 con 0x7f298c0ffe50 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f2991afb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c19e620 con 0x7f298c1007f0 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f298c104bc0 0x7f298c19dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f298c104bc0 0x7f298c19dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37354/0 (socket says 172.21.15.136:37354) 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298bfff640 1 -- 172.21.15.136:0/281774169 learned_addr learned my addr 172.21.15.136:0/281774169 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298affd640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c1007f0 0x7f298c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298b7fe640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f298c0ffe50 0x7f298c199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298bfff640 1 -- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c1007f0 msgr2=0x7f298c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298bfff640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c1007f0 0x7f298c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298bfff640 1 -- 172.21.15.136:0/281774169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f298c0ffe50 msgr2=0x7f298c199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298bfff640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f298c0ffe50 0x7f298c199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:12.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298bfff640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2984009840 con 0x7f298c104bc0 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.853+0000 7f298affd640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c1007f0 0x7f298c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.854+0000 7f298bfff640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f298c104bc0 0x7f298c19dcf0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f298400efd0 tx=0x7f298400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.854+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984009b40 con 0x7f298c104bc0 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.854+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f2984003b70 con 0x7f298c104bc0 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.854+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2980009560 con 0x7f298c104bc0 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.854+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984010840 con 0x7f298c104bc0 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.854+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c1a5fa0 con 0x7f298c104bc0 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.855+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298c0688c0 con 0x7f298c104bc0 2024-03-26T05:56:12.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.855+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f2984004020 con 0x7f298c104bc0 2024-03-26T05:56:12.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.855+0000 7f2988ff9640 1 --2- 172.21.15.136:0/281774169 >> v2:172.21.15.136:6800/3380174081 conn(0x7f297406d9f0 0x7f297406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:12.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.856+0000 7f298b7fe640 1 --2- 172.21.15.136:0/281774169 >> v2:172.21.15.136:6800/3380174081 conn(0x7f297406d9f0 0x7f297406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:12.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.856+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(817..817 src has 263..817) v4 ==== 7762+0+0 (secure 0 0 0) 0x7f2984014070 con 0x7f298c104bc0 2024-03-26T05:56:12.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.856+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=818}) v3 -- 0x7f2974078e50 con 0x7f298c104bc0 2024-03-26T05:56:12.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.856+0000 7f298b7fe640 1 --2- 172.21.15.136:0/281774169 >> v2:172.21.15.136:6800/3380174081 conn(0x7f297406d9f0 0x7f297406feb0 secure :-1 s=READY pgs=3549 cs=0 l=1 rev1=1 crypto rx=0x7f297c00ba40 tx=0x7f297c005cc0 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:12.852 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:12.860+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f298405ae40 con 0x7f298c104bc0 2024-03-26T05:56:13.029 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:13.037+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f298c10fce0 con 0x7f298c104bc0 2024-03-26T05:56:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:56:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:12 smithi136 ceph-mon[38106]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T05:56:13.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:12 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2433668571' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:13.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:12 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:56:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:12 smithi136 ceph-mon[29560]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T05:56:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:12 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2433668571' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:13.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:12 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:13.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:56:13.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:12 smithi164 ceph-mon[33554]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T05:56:13.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:12 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2433668571' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:13.120 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:12 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:56:13.736 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:13.743+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v818) v1 ==== 221+0+0 (secure 0 0 0) 0x7f298405a9d0 con 0x7f298c104bc0 2024-03-26T05:56:13.736 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:13.743+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 8 ==== osd_map(818..818 src has 263..818) v4 ==== 628+0+0 (secure 0 0 0) 0x7f298405b770 con 0x7f298c104bc0 2024-03-26T05:56:13.736 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:13.744+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=819}) v3 -- 0x7f2974079df0 con 0x7f298c104bc0 2024-03-26T05:56:13.847 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:13.855+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f298c1a6130 con 0x7f298c104bc0 2024-03-26T05:56:13.969 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:56:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:56:13.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:13 smithi164 ceph-mon[33554]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:56:13.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:13 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:56:13.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:13 smithi164 ceph-mon[33554]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T05:56:13.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:13 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:56:13.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:14.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[38106]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[38106]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[29560]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[29560]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:56:14.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:14.248 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.255+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v819) v1 ==== 221+0+0 (secure 0 0 0) 0x7f298405abb0 con 0x7f298c104bc0 2024-03-26T05:56:14.248 INFO:tasks.workunit.client.0.smithi136.stderr:set-quota max_objects = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 2024-03-26T05:56:14.249 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.257+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 <== mon.0 v2:172.21.15.136:3300/0 10 ==== osd_map(819..819 src has 263..819) v4 ==== 628+0+0 (secure 0 0 0) 0x7f298405f1c0 con 0x7f298c104bc0 2024-03-26T05:56:14.250 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.257+0000 7f2988ff9640 1 -- 172.21.15.136:0/281774169 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=820}) v3 -- 0x7f2974079fc0 con 0x7f298c104bc0 2024-03-26T05:56:14.251 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 >> v2:172.21.15.136:6800/3380174081 conn(0x7f297406d9f0 msgr2=0x7f297406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:14.251 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 --2- 172.21.15.136:0/281774169 >> v2:172.21.15.136:6800/3380174081 conn(0x7f297406d9f0 0x7f297406feb0 secure :-1 s=READY pgs=3549 cs=0 l=1 rev1=1 crypto rx=0x7f297c00ba40 tx=0x7f297c005cc0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.251 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f298c104bc0 msgr2=0x7f298c19dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:14.251 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f298c104bc0 0x7f298c19dcf0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f298400efd0 tx=0x7f298400c6a0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.251 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 shutdown_connections 2024-03-26T05:56:14.251 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 --2- 172.21.15.136:0/281774169 >> v2:172.21.15.136:6800/3380174081 conn(0x7f297406d9f0 0x7f297406feb0 unknown :-1 s=CLOSED pgs=3549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.252 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f298c104bc0 0x7f298c19dcf0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.252 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f298c1007f0 0x7f298c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.252 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 --2- 172.21.15.136:0/281774169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f298c0ffe50 0x7f298c199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.252 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.259+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 >> 172.21.15.136:0/281774169 conn(0x7f298c0757b0 msgr2=0x7f298c10b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:14.254 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.262+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 shutdown_connections 2024-03-26T05:56:14.255 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.263+0000 7f2991afb640 1 -- 172.21.15.136:0/281774169 wait complete. 2024-03-26T05:56:14.267 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait 151866 2024-03-26T05:56:14.267 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T05:56:14.267 INFO:tasks.workunit.client.0.smithi136.stderr:+ true 2024-03-26T05:56:14.267 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put three /etc/passwd 2024-03-26T05:56:14.304 INFO:tasks.workunit.client.0.smithi136.stderr:++ uuidgen 2024-03-26T05:56:14.305 INFO:tasks.workunit.client.0.smithi136.stderr:+ pp=e5c100e4-1040-435b-858b-cfd5917b91be 2024-03-26T05:56:14.305 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool create e5c100e4-1040-435b-858b-cfd5917b91be 12 2024-03-26T05:56:14.390 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.395+0000 7f5f68565640 1 -- 172.21.15.136:0/3275461907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100e10 msgr2=0x7f5f6010d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.395+0000 7f5f68565640 1 --2- 172.21.15.136:0/3275461907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100e10 0x7f5f6010d320 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009c10 tx=0x7f5f5802bef0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 -- 172.21.15.136:0/3275461907 shutdown_connections 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 --2- 172.21.15.136:0/3275461907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100e10 0x7f5f6010d320 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 --2- 172.21.15.136:0/3275461907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f60100450 0x7f5f601008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 --2- 172.21.15.136:0/3275461907 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f601064f0 0x7f5f601068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 -- 172.21.15.136:0/3275461907 >> 172.21.15.136:0/3275461907 conn(0x7f5f600775b0 msgr2=0x7f5f600779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 -- 172.21.15.136:0/3275461907 shutdown_connections 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 -- 172.21.15.136:0/3275461907 wait complete. 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 Processor -- start 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 -- start start 2024-03-26T05:56:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100450 0x7f5f60197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f60100e10 0x7f5f60197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f601064f0 0x7f5f6019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6019c170 con 0x7f5f60100450 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6019c2e0 con 0x7f5f601064f0 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f68565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6019c4a0 con 0x7f5f60100e10 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f65b00640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f60100e10 0x7f5f60197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.396+0000 7f5f66301640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100450 0x7f5f60197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f65b00640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f60100e10 0x7f5f60197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:52856/0 (socket says 172.21.15.136:52856) 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f65b00640 1 -- 172.21.15.136:0/139763223 learned_addr learned my addr 172.21.15.136:0/139763223 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f66301640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100450 0x7f5f60197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37406/0 (socket says 172.21.15.136:37406) 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f66b02640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f601064f0 0x7f5f6019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f65b00640 1 -- 172.21.15.136:0/139763223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f601064f0 msgr2=0x7f5f6019bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f65b00640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f601064f0 0x7f5f6019bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f65b00640 1 -- 172.21.15.136:0/139763223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100450 msgr2=0x7f5f60197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f65b00640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100450 0x7f5f60197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f65b00640 1 -- 172.21.15.136:0/139763223 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c009560 con 0x7f5f60100e10 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f66301640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100450 0x7f5f60197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f65b00640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f60100e10 0x7f5f60197840 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000d950 tx=0x7f5f5000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f577fe640 1 -- 172.21.15.136:0/139763223 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50009c30 con 0x7f5f60100e10 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f577fe640 1 -- 172.21.15.136:0/139763223 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5f50009d90 con 0x7f5f60100e10 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f58009840 con 0x7f5f60100e10 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f577fe640 1 -- 172.21.15.136:0/139763223 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5000f690 con 0x7f5f60100e10 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.397+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f601a3e20 con 0x7f5f60100e10 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.398+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f60068470 con 0x7f5f60100e10 2024-03-26T05:56:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.399+0000 7f5f577fe640 1 -- 172.21.15.136:0/139763223 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f5f50010460 con 0x7f5f60100e10 2024-03-26T05:56:14.394 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.399+0000 7f5f577fe640 1 --2- 172.21.15.136:0/139763223 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5f4c06d9f0 0x7f5f4c06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:14.394 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.399+0000 7f5f577fe640 1 -- 172.21.15.136:0/139763223 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(819..819 src has 263..819) v4 ==== 7762+0+0 (secure 0 0 0) 0x7f5f5008d610 con 0x7f5f60100e10 2024-03-26T05:56:14.394 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.399+0000 7f5f66301640 1 --2- 172.21.15.136:0/139763223 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5f4c06d9f0 0x7f5f4c06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:14.394 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.399+0000 7f5f66301640 1 --2- 172.21.15.136:0/139763223 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5f4c06d9f0 0x7f5f4c06feb0 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009530 tx=0x7f5f5c009260 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:14.396 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.404+0000 7f5f577fe640 1 -- 172.21.15.136:0/139763223 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5f5005a500 con 0x7f5f60100e10 2024-03-26T05:56:14.570 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:14.578+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12} v 0) v1 -- 0x7f5f6010fc30 con 0x7f5f60100e10 2024-03-26T05:56:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:56:15.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[38106]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[38106]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[38106]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[38106]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[29560]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[29560]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:56:15.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[29560]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:56:15.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:56:15.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:14 smithi136 ceph-mon[29560]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T05:56:15.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:15.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:56:15.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:14 smithi164 ceph-mon[33554]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T05:56:15.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:56:15.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:14 smithi164 ceph-mon[33554]: pool '78bcab27-41f1-4625-8adb-19d91c406d36' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:56:15.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:14 smithi164 ceph-mon[33554]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:56:15.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:14 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/281774169' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:56:15.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:14 smithi164 ceph-mon[33554]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T05:56:15.258 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.264+0000 7f5f577fe640 1 -- 172.21.15.136:0/139763223 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]=0 pool 'e5c100e4-1040-435b-858b-cfd5917b91be' created v820) v1 ==== 176+0+0 (secure 0 0 0) 0x7f5f5005a090 con 0x7f5f60100e10 2024-03-26T05:56:15.366 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.373+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12} v 0) v1 -- 0x7f5f6010ff20 con 0x7f5f60100e10 2024-03-26T05:56:15.367 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.374+0000 7f5f577fe640 1 -- 172.21.15.136:0/139763223 <== mon.2 v2:172.21.15.136:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]=0 pool 'e5c100e4-1040-435b-858b-cfd5917b91be' already exists v820) v1 ==== 183+0+0 (secure 0 0 0) 0x7f5f50017070 con 0x7f5f60100e10 2024-03-26T05:56:15.367 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'e5c100e4-1040-435b-858b-cfd5917b91be' already exists 2024-03-26T05:56:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5f4c06d9f0 msgr2=0x7f5f4c06feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 --2- 172.21.15.136:0/139763223 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5f4c06d9f0 0x7f5f4c06feb0 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009530 tx=0x7f5f5c009260 comp rx=0 tx=0).stop 2024-03-26T05:56:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f60100e10 msgr2=0x7f5f60197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f60100e10 0x7f5f60197840 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000d950 tx=0x7f5f5000de10 comp rx=0 tx=0).stop 2024-03-26T05:56:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 shutdown_connections 2024-03-26T05:56:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 --2- 172.21.15.136:0/139763223 >> v2:172.21.15.136:6800/3380174081 conn(0x7f5f4c06d9f0 0x7f5f4c06feb0 unknown :-1 s=CLOSED pgs=3551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f601064f0 0x7f5f6019bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f60100e10 0x7f5f60197840 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 --2- 172.21.15.136:0/139763223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f60100450 0x7f5f60197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 >> 172.21.15.136:0/139763223 conn(0x7f5f600775b0 msgr2=0x7f5f600688c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:15.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 shutdown_connections 2024-03-26T05:56:15.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.376+0000 7f5f68565640 1 -- 172.21.15.136:0/139763223 wait complete. 2024-03-26T05:56:15.378 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool application enable e5c100e4-1040-435b-858b-cfd5917b91be rados 2024-03-26T05:56:15.463 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 -- 172.21.15.136:0/129506718 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8107640 msgr2=0x7f84c8107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:15.463 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/129506718 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8107640 0x7f84c8107a20 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009890 tx=0x7f84b802bb70 comp rx=0 tx=0).stop 2024-03-26T05:56:15.463 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 -- 172.21.15.136:0/129506718 shutdown_connections 2024-03-26T05:56:15.463 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/129506718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84c8101f40 0x7f84c810e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/129506718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84c8101580 0x7f84c8101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/129506718 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8107640 0x7f84c8107a20 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 -- 172.21.15.136:0/129506718 >> 172.21.15.136:0/129506718 conn(0x7f84c80fd070 msgr2=0x7f84c80ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 -- 172.21.15.136:0/129506718 shutdown_connections 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 -- 172.21.15.136:0/129506718 wait complete. 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 Processor -- start 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 -- start start 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.468+0000 7f84cdd2e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84c8101580 0x7f84c81972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84cdd2e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8101f40 0x7f84c8197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84cdd2e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84c8107640 0x7f84c819bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84cdd2e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c819c190 con 0x7f84c8107640 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84cdd2e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c819c300 con 0x7f84c8101580 2024-03-26T05:56:15.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84cdd2e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c819c4c0 con 0x7f84c8101f40 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8101f40 0x7f84c8197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84c7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84c8107640 0x7f84c819bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8101f40 0x7f84c8197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:52882/0 (socket says 172.21.15.136:52882) 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 -- 172.21.15.136:0/3393804581 learned_addr learned my addr 172.21.15.136:0/3393804581 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84c77fe640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84c8101580 0x7f84c81972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 -- 172.21.15.136:0/3393804581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84c8101580 msgr2=0x7f84c81972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84c8101580 0x7f84c81972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 -- 172.21.15.136:0/3393804581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84c8107640 msgr2=0x7f84c819bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84c8107640 0x7f84c819bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 -- 172.21.15.136:0/3393804581 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0009560 con 0x7f84c8101f40 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84c77fe640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84c8101580 0x7f84c81972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84c7fff640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84c8107640 0x7f84c819bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:56:15.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84beffd640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8101f40 0x7f84c8197810 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f84b000e9c0 tx=0x7f84b000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:15.466 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84c57fa640 1 -- 172.21.15.136:0/3393804581 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b000cde0 con 0x7f84c8101f40 2024-03-26T05:56:15.466 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84c57fa640 1 -- 172.21.15.136:0/3393804581 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f84b00044d0 con 0x7f84c8101f40 2024-03-26T05:56:15.466 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b8009490 con 0x7f84c8101f40 2024-03-26T05:56:15.466 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84c57fa640 1 -- 172.21.15.136:0/3393804581 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b0010430 con 0x7f84c8101f40 2024-03-26T05:56:15.466 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.469+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c81a3e40 con 0x7f84c8101f40 2024-03-26T05:56:15.466 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.470+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c80688c0 con 0x7f84c8101f40 2024-03-26T05:56:15.467 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.471+0000 7f84c57fa640 1 -- 172.21.15.136:0/3393804581 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f84b00035f0 con 0x7f84c8101f40 2024-03-26T05:56:15.467 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.471+0000 7f84c57fa640 1 --2- 172.21.15.136:0/3393804581 >> v2:172.21.15.136:6800/3380174081 conn(0x7f84b406d9f0 0x7f84b406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:15.467 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.471+0000 7f84c77fe640 1 --2- 172.21.15.136:0/3393804581 >> v2:172.21.15.136:6800/3380174081 conn(0x7f84b406d9f0 0x7f84b406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:15.467 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.471+0000 7f84c57fa640 1 -- 172.21.15.136:0/3393804581 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(820..820 src has 263..820) v4 ==== 8137+0+0 (secure 0 0 0) 0x7f84b0014070 con 0x7f84c8101f40 2024-03-26T05:56:15.467 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.472+0000 7f84c77fe640 1 --2- 172.21.15.136:0/3393804581 >> v2:172.21.15.136:6800/3380174081 conn(0x7f84b406d9f0 0x7f84b406feb0 secure :-1 s=READY pgs=3552 cs=0 l=1 rev1=1 crypto rx=0x7f84b80323d0 tx=0x7f84b8032340 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:15.469 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.476+0000 7f84c57fa640 1 -- 172.21.15.136:0/3393804581 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f84b005aca0 con 0x7f84c8101f40 2024-03-26T05:56:15.643 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:15.650+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"} v 0) v1 -- 0x7f84c8110d60 con 0x7f84c8101f40 2024-03-26T05:56:16.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/139763223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]': finished 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/139763223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3393804581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/139763223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]': finished 2024-03-26T05:56:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T05:56:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/139763223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3393804581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:15 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/139763223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]': finished 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/139763223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pg_num": 12}]: dispatch 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3393804581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:15 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:16.266+0000 7f84c57fa640 1 -- 172.21.15.136:0/3393804581 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]=0 enabled application 'rados' on pool 'e5c100e4-1040-435b-858b-cfd5917b91be' v821) v1 ==== 213+0+0 (secure 0 0 0) 0x7f84b005a830 con 0x7f84c8101f40 2024-03-26T05:56:16.375 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:16.381+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"} v 0) v1 -- 0x7f84c8110ef0 con 0x7f84c8101f40 2024-03-26T05:56:17.267 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.275+0000 7f84c57fa640 1 -- 172.21.15.136:0/3393804581 <== mon.2 v2:172.21.15.136:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]=0 enabled application 'rados' on pool 'e5c100e4-1040-435b-858b-cfd5917b91be' v822) v1 ==== 213+0+0 (secure 0 0 0) 0x7f84b005aa10 con 0x7f84c8101f40 2024-03-26T05:56:17.268 INFO:tasks.workunit.client.0.smithi136.stderr:enabled application 'rados' on pool 'e5c100e4-1040-435b-858b-cfd5917b91be' 2024-03-26T05:56:17.270 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 >> v2:172.21.15.136:6800/3380174081 conn(0x7f84b406d9f0 msgr2=0x7f84b406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:17.270 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/3393804581 >> v2:172.21.15.136:6800/3380174081 conn(0x7f84b406d9f0 0x7f84b406feb0 secure :-1 s=READY pgs=3552 cs=0 l=1 rev1=1 crypto rx=0x7f84b80323d0 tx=0x7f84b8032340 comp rx=0 tx=0).stop 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8101f40 msgr2=0x7f84c8197810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8101f40 0x7f84c8197810 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f84b000e9c0 tx=0x7f84b000ee80 comp rx=0 tx=0).stop 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 shutdown_connections 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/3393804581 >> v2:172.21.15.136:6800/3380174081 conn(0x7f84b406d9f0 0x7f84b406feb0 unknown :-1 s=CLOSED pgs=3552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84c8107640 0x7f84c819bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84c8101f40 0x7f84c8197810 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 --2- 172.21.15.136:0/3393804581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84c8101580 0x7f84c81972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 >> 172.21.15.136:0/3393804581 conn(0x7f84c80fd070 msgr2=0x7f84c80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 shutdown_connections 2024-03-26T05:56:17.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.278+0000 7f84cdd2e640 1 -- 172.21.15.136:0/3393804581 wait complete. 2024-03-26T05:56:17.281 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set-quota e5c100e4-1040-435b-858b-cfd5917b91be max_objects 10 2024-03-26T05:56:17.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.376+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/324047048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 msgr2=0x7f3ba010e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:17.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.376+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/324047048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 0x7f3ba010e400 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009890 tx=0x7f3b9802bb70 comp rx=0 tx=0).stop 2024-03-26T05:56:17.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/324047048 shutdown_connections 2024-03-26T05:56:17.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/324047048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 0x7f3ba010e400 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/324047048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ba0101580 0x7f3ba0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/324047048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba0107640 0x7f3ba0107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/324047048 >> 172.21.15.136:0/324047048 conn(0x7f3ba00fd090 msgr2=0x7f3ba00ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:17.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/324047048 shutdown_connections 2024-03-26T05:56:17.370 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/324047048 wait complete. 2024-03-26T05:56:17.372 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 Processor -- start 2024-03-26T05:56:17.372 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 -- start start 2024-03-26T05:56:17.372 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.377+0000 7f3ba4dc7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba0101580 0x7f3ba01971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:17.372 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3ba4dc7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 0x7f3ba01976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3ba4dc7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ba0107640 0x7f3ba019ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3ba4dc7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba019c070 con 0x7f3ba0101580 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3ba4dc7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba019c1e0 con 0x7f3ba0107640 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3ba4dc7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba019c3a0 con 0x7f3ba0101f40 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba0101580 0x7f3ba01971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9dd9b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 0x7f3ba01976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba0101580 0x7f3ba01971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37440/0 (socket says 172.21.15.136:37440) 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9e59c640 1 -- 172.21.15.136:0/2861227307 learned_addr learned my addr 172.21.15.136:0/2861227307 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9ed9d640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ba0107640 0x7f3ba019ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9dd9b640 1 -- 172.21.15.136:0/2861227307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ba0107640 msgr2=0x7f3ba019ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9dd9b640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ba0107640 0x7f3ba019ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9dd9b640 1 -- 172.21.15.136:0/2861227307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba0101580 msgr2=0x7f3ba01971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9dd9b640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba0101580 0x7f3ba01971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9dd9b640 1 -- 172.21.15.136:0/2861227307 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b90009900 con 0x7f3ba0101f40 2024-03-26T05:56:17.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9e59c640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba0101580 0x7f3ba01971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b9dd9b640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 0x7f3ba01976f0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800c970 tx=0x7f3b8800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b977fe640 1 -- 172.21.15.136:0/2861227307 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88007bb0 con 0x7f3ba0101f40 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b98009490 con 0x7f3ba0101f40 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b977fe640 1 -- 172.21.15.136:0/2861227307 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3b88007d10 con 0x7f3ba0101f40 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3b977fe640 1 -- 172.21.15.136:0/2861227307 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88018660 con 0x7f3ba0101f40 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.378+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba01a3d20 con 0x7f3ba0101f40 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.379+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba00688c0 con 0x7f3ba0101f40 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.380+0000 7f3b977fe640 1 -- 172.21.15.136:0/2861227307 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f3b8800f450 con 0x7f3ba0101f40 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.380+0000 7f3b977fe640 1 --2- 172.21.15.136:0/2861227307 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3b8c06d9f0 0x7f3b8c06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.380+0000 7f3b977fe640 1 -- 172.21.15.136:0/2861227307 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(822..822 src has 263..822) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f3b8808dc00 con 0x7f3ba0101f40 2024-03-26T05:56:17.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.380+0000 7f3b9e59c640 1 --2- 172.21.15.136:0/2861227307 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3b8c06d9f0 0x7f3b8c06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:56:17.375 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.381+0000 7f3b9e59c640 1 --2- 172.21.15.136:0/2861227307 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3b8c06d9f0 0x7f3b8c06feb0 secure :-1 s=READY pgs=3553 cs=0 l=1 rev1=1 crypto rx=0x7f3b9000e9d0 tx=0x7f3b900095c0 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:56:17.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.385+0000 7f3b977fe640 1 -- 172.21.15.136:0/2861227307 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3b88094080 con 0x7f3ba0101f40 2024-03-26T05:56:17.554 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:17.561+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f3ba0110d80 con 0x7f3ba0101f40 2024-03-26T05:56:17.575 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:56:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]': finished 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[38106]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3393804581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[38106]: pgmap v1775: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.4 KiB/s wr, 1 op/s 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]': finished 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[29560]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3393804581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:17.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:17 smithi136 ceph-mon[29560]: pgmap v1775: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.4 KiB/s wr, 1 op/s 2024-03-26T05:56:17.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]': finished 2024-03-26T05:56:17.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:17 smithi164 ceph-mon[33554]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T05:56:17.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:17 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3393804581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:17.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:17 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]: dispatch 2024-03-26T05:56:17.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:17 smithi164 ceph-mon[33554]: pgmap v1775: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.4 KiB/s wr, 1 op/s 2024-03-26T05:56:18.268 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:18.275+0000 7f3b977fe640 1 -- 172.21.15.136:0/2861227307 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool e5c100e4-1040-435b-858b-cfd5917b91be v823) v1 ==== 223+0+0 (secure 0 0 0) 0x7f3b8805a8c0 con 0x7f3ba0101f40 2024-03-26T05:56:18.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:18.385+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f3ba0198500 con 0x7f3ba0101f40 2024-03-26T05:56:18.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]': finished 2024-03-26T05:56:18.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:18 smithi136 ceph-mon[38106]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T05:56:18.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:18 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2861227307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:18.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:18 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:18.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]': finished 2024-03-26T05:56:18.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:18 smithi136 ceph-mon[29560]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T05:56:18.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:18 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2861227307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:18.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:18 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "app": "rados"}]': finished 2024-03-26T05:56:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:18 smithi164 ceph-mon[33554]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T05:56:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:18 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2861227307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:18.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:18 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:19.281 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.288+0000 7f3b977fe640 1 -- 172.21.15.136:0/2861227307 <== mon.2 v2:172.21.15.136:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool e5c100e4-1040-435b-858b-cfd5917b91be v824) v1 ==== 223+0+0 (secure 0 0 0) 0x7f3b8805a450 con 0x7f3ba0101f40 2024-03-26T05:56:19.281 INFO:tasks.workunit.client.0.smithi136.stderr:set-quota max_objects = 10 for pool e5c100e4-1040-435b-858b-cfd5917b91be 2024-03-26T05:56:19.284 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3b8c06d9f0 msgr2=0x7f3b8c06feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:19.284 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/2861227307 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3b8c06d9f0 0x7f3b8c06feb0 secure :-1 s=READY pgs=3553 cs=0 l=1 rev1=1 crypto rx=0x7f3b9000e9d0 tx=0x7f3b900095c0 comp rx=0 tx=0).stop 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 msgr2=0x7f3ba01976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 0x7f3ba01976f0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800c970 tx=0x7f3b8800ce30 comp rx=0 tx=0).stop 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 shutdown_connections 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/2861227307 >> v2:172.21.15.136:6800/3380174081 conn(0x7f3b8c06d9f0 0x7f3b8c06feb0 unknown :-1 s=CLOSED pgs=3553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ba0107640 0x7f3ba019ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba0101f40 0x7f3ba01976f0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 --2- 172.21.15.136:0/2861227307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba0101580 0x7f3ba01971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 >> 172.21.15.136:0/2861227307 conn(0x7f3ba00fd090 msgr2=0x7f3ba0105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 shutdown_connections 2024-03-26T05:56:19.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:56:19.292+0000 7f3ba4dc7640 1 -- 172.21.15.136:0/2861227307 wait complete. 2024-03-26T05:56:19.296 INFO:tasks.workunit.client.0.smithi136.stderr:+ sleep 30 2024-03-26T05:56:19.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[38106]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/2861227307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[38106]: pgmap v1778: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[29560]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/2861227307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:19.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:19 smithi136 ceph-mon[29560]: pgmap v1778: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T05:56:19.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:56:19.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:19 smithi164 ceph-mon[33554]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T05:56:19.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:19 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/2861227307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:19.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:19 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:56:19.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:19 smithi164 ceph-mon[33554]: pgmap v1778: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T05:56:20.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:20 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:56:20.576 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:20 smithi136 ceph-mon[29560]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T05:56:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:20 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:56:20.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:20 smithi136 ceph-mon[38106]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T05:56:20.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:20 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:56:20.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:20 smithi164 ceph-mon[33554]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T05:56:21.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:21 smithi136 ceph-mon[29560]: pgmap v1780: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:56:21.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:21 smithi136 ceph-mon[38106]: pgmap v1780: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:56:21.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:21 smithi164 ceph-mon[33554]: pgmap v1780: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:56:23.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:23 smithi136 ceph-mon[38106]: pgmap v1781: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:23.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:23 smithi136 ceph-mon[29560]: pgmap v1781: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:23.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:23 smithi164 ceph-mon[33554]: pgmap v1781: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:23.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:56:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:56:24.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:24.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:24.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:25.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:25 smithi136 ceph-mon[38106]: pgmap v1782: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-26T05:56:25.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:25 smithi136 ceph-mon[29560]: pgmap v1782: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-26T05:56:25.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:25 smithi164 ceph-mon[33554]: pgmap v1782: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-26T05:56:27.566 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:56:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:56:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:56:27.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:27 smithi136 ceph-mon[38106]: pgmap v1783: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:27.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:27 smithi136 ceph-mon[29560]: pgmap v1783: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:27.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:27 smithi164 ceph-mon[33554]: pgmap v1783: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:30.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:29 smithi136 ceph-mon[38106]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:56:30.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:29 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:56:30.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:29 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:30.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:29 smithi136 ceph-mon[29560]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:56:30.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:29 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:56:30.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:29 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:30.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:29 smithi164 ceph-mon[33554]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:56:30.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:29 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:56:30.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:29 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:32.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:31 smithi136 ceph-mon[38106]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-26T05:56:32.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:31 smithi136 ceph-mon[29560]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-26T05:56:32.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:31 smithi164 ceph-mon[33554]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-26T05:56:33.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:32 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:56:33.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:32 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:56:33.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:32 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:56:33.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:56:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:56:33.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:33 smithi164 ceph-mon[33554]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:33.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:56:33.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:56:33.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:33 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:56:34.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:33 smithi136 ceph-mon[38106]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:34.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:56:34.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:56:34.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:33 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:56:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:33 smithi136 ceph-mon[29560]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:56:34.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:56:34.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:33 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:56:35.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:35.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:35.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:36.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:35 smithi136 ceph-mon[38106]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:36.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:35 smithi136 ceph-mon[29560]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:36.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:35 smithi164 ceph-mon[33554]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:37.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:56:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:56:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:56:38.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:37 smithi136 ceph-mon[38106]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:38.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:37 smithi136 ceph-mon[29560]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:38.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:37 smithi164 ceph-mon[33554]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:40.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:39 smithi136 ceph-mon[38106]: pgmap v1789: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:40.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:39 smithi136 ceph-mon[29560]: pgmap v1789: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:40.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:39 smithi164 ceph-mon[33554]: pgmap v1789: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:42.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:41 smithi136 ceph-mon[38106]: pgmap v1790: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:42.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:41 smithi136 ceph-mon[29560]: pgmap v1790: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:42.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:41 smithi164 ceph-mon[33554]: pgmap v1790: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:43.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:43 smithi164 ceph-mon[33554]: pgmap v1791: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:43.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:43.971 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:56:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:56:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:43 smithi136 ceph-mon[38106]: pgmap v1791: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:44.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:43 smithi136 ceph-mon[29560]: pgmap v1791: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:44.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:56:45.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:45.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:45.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:46.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:45 smithi136 ceph-mon[38106]: pgmap v1792: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:46.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:45 smithi136 ceph-mon[29560]: pgmap v1792: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:46.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:45 smithi164 ceph-mon[33554]: pgmap v1792: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:47.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:56:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:56:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:56:48.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:47 smithi136 ceph-mon[38106]: pgmap v1793: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:48.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:47 smithi136 ceph-mon[29560]: pgmap v1793: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:48.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:47 smithi164 ceph-mon[33554]: pgmap v1793: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:56:49.298 INFO:tasks.workunit.client.0.smithi136.stderr:++ seq 1 10 2024-03-26T05:56:49.299 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.299 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj1 /etc/passwd 2024-03-26T05:56:49.340 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.340 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj2 /etc/passwd 2024-03-26T05:56:49.377 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.377 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj3 /etc/passwd 2024-03-26T05:56:49.413 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.413 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj4 /etc/passwd 2024-03-26T05:56:49.448 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.448 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj5 /etc/passwd 2024-03-26T05:56:49.482 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.483 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj6 /etc/passwd 2024-03-26T05:56:49.517 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.517 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj7 /etc/passwd 2024-03-26T05:56:49.552 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.552 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj8 /etc/passwd 2024-03-26T05:56:49.586 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.586 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj9 /etc/passwd 2024-03-26T05:56:49.621 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in `seq 1 10` 2024-03-26T05:56:49.621 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p e5c100e4-1040-435b-858b-cfd5917b91be put obj10 /etc/passwd 2024-03-26T05:56:49.655 INFO:tasks.workunit.client.0.smithi136.stderr:+ sleep 30 2024-03-26T05:56:50.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:49 smithi136 ceph-mon[38106]: pgmap v1794: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:50.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:49 smithi136 ceph-mon[29560]: pgmap v1794: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:50.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:49 smithi164 ceph-mon[33554]: pgmap v1794: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:56:52.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:51 smithi136 ceph-mon[38106]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:52.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:51 smithi136 ceph-mon[29560]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:52.222 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:51 smithi164 ceph-mon[33554]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:56:53.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:53 smithi164 ceph-mon[33554]: pgmap v1796: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T05:56:53.971 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:56:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:56:54.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:53 smithi136 ceph-mon[38106]: pgmap v1796: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T05:56:54.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:53 smithi136 ceph-mon[29560]: pgmap v1796: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T05:56:55.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:55.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:54 smithi136 ceph-mon[38106]: pool 'e5c100e4-1040-435b-858b-cfd5917b91be' is full (reached quota's max_objects: 10) 2024-03-26T05:56:55.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:54 smithi136 ceph-mon[38106]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:56:55.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:54 smithi136 ceph-mon[38106]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T05:56:55.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:55.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:54 smithi136 ceph-mon[29560]: pool 'e5c100e4-1040-435b-858b-cfd5917b91be' is full (reached quota's max_objects: 10) 2024-03-26T05:56:55.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:54 smithi136 ceph-mon[29560]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:56:55.079 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:54 smithi136 ceph-mon[29560]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T05:56:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:56:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:54 smithi164 ceph-mon[33554]: pool 'e5c100e4-1040-435b-858b-cfd5917b91be' is full (reached quota's max_objects: 10) 2024-03-26T05:56:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:54 smithi164 ceph-mon[33554]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:56:55.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:54 smithi164 ceph-mon[33554]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T05:56:56.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:55 smithi136 ceph-mon[29560]: pgmap v1798: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:56:56.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:55 smithi136 ceph-mon[38106]: pgmap v1798: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:56:56.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:55 smithi164 ceph-mon[33554]: pgmap v1798: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:56:57.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:56:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:56:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:56:58.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:57 smithi136 ceph-mon[38106]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:56:58.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:57 smithi136 ceph-mon[29560]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:56:58.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:57 smithi164 ceph-mon[33554]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:57:00.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:59 smithi136 ceph-mon[29560]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:57:00.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:59 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:00.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:56:59 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:00.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:59 smithi136 ceph-mon[38106]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:57:00.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:59 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:00.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:56:59 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:00.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:59 smithi164 ceph-mon[33554]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:57:00.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:59 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:00.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:56:59 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:02.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:01 smithi136 ceph-mon[29560]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:57:02.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:01 smithi136 ceph-mon[38106]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:57:02.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:01 smithi164 ceph-mon[33554]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:57:03.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:03 smithi164 ceph-mon[33554]: pgmap v1802: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:57:03.971 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:57:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:57:04.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:03 smithi136 ceph-mon[29560]: pgmap v1802: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:57:04.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:03 smithi136 ceph-mon[38106]: pgmap v1802: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:57:05.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:04 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:05.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:04 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:05.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:04 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:06.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:05 smithi136 ceph-mon[38106]: pgmap v1803: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-26T05:57:06.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:05 smithi136 ceph-mon[29560]: pgmap v1803: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-26T05:57:06.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:05 smithi164 ceph-mon[33554]: pgmap v1803: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-26T05:57:07.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:57:07 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:57:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:57:08.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:07 smithi136 ceph-mon[38106]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:08.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:07 smithi136 ceph-mon[29560]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:08.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:07 smithi164 ceph-mon[33554]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:10.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:09 smithi136 ceph-mon[29560]: pgmap v1805: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:10.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:09 smithi136 ceph-mon[38106]: pgmap v1805: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:10.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:09 smithi164 ceph-mon[33554]: pgmap v1805: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:12.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:11 smithi136 ceph-mon[38106]: pgmap v1806: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:12.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:11 smithi136 ceph-mon[29560]: pgmap v1806: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:12.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:11 smithi164 ceph-mon[33554]: pgmap v1806: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:14.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:13 smithi164 ceph-mon[33554]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:14.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:13 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:14.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:57:13 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:57:14.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:13 smithi136 ceph-mon[38106]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:14.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:13 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:14.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:13 smithi136 ceph-mon[29560]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:14.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:13 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:15.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:14 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:15.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:14 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:15.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:14 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:16.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:15 smithi136 ceph-mon[38106]: pgmap v1808: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:16.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:15 smithi136 ceph-mon[29560]: pgmap v1808: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:16.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:15 smithi164 ceph-mon[33554]: pgmap v1808: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:17.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:57:17 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:57:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:57:18.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:17 smithi136 ceph-mon[38106]: pgmap v1809: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:18.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:17 smithi136 ceph-mon[29560]: pgmap v1809: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:18.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:17 smithi164 ceph-mon[33554]: pgmap v1809: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:19.657 INFO:tasks.workunit.client.0.smithi136.stderr:+ rados -p 78bcab27-41f1-4625-8adb-19d91c406d36 put threemore /etc/passwd 2024-03-26T05:57:19.698 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set-quota 78bcab27-41f1-4625-8adb-19d91c406d36 max_bytes 0 2024-03-26T05:57:19.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 -- 172.21.15.136:0/612897453 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4910100380 msgr2=0x7f4910100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:19.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 --2- 172.21.15.136:0/612897453 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4910100380 0x7f4910100760 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f4900009900 tx=0x7f490002bc30 comp rx=0 tx=0).stop 2024-03-26T05:57:19.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 -- 172.21.15.136:0/612897453 shutdown_connections 2024-03-26T05:57:19.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 --2- 172.21.15.136:0/612897453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f491010df30 0x7f4910110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:19.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 --2- 172.21.15.136:0/612897453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4910100ca0 0x7f491010d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:19.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 --2- 172.21.15.136:0/612897453 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4910100380 0x7f4910100760 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:19.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 -- 172.21.15.136:0/612897453 >> 172.21.15.136:0/612897453 conn(0x7f4910077670 msgr2=0x7f4910077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:57:19.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 -- 172.21.15.136:0/612897453 shutdown_connections 2024-03-26T05:57:19.787 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.792+0000 7f49153c7640 1 -- 172.21.15.136:0/612897453 wait complete. 2024-03-26T05:57:19.787 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f49153c7640 1 Processor -- start 2024-03-26T05:57:19.787 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f49153c7640 1 -- start start 2024-03-26T05:57:19.789 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f49153c7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4910100380 0x7f4910199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:19.789 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f49153c7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910100ca0 0x7f4910199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:19.789 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f49153c7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f491010df30 0x7f491019de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:19.789 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f49153c7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491019e410 con 0x7f4910100ca0 2024-03-26T05:57:19.789 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f49153c7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491019e580 con 0x7f4910100380 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f49153c7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491019e740 con 0x7f491010df30 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f490e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910100ca0 0x7f4910199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f490f7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f491010df30 0x7f491019de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f490e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910100ca0 0x7f4910199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:42840/0 (socket says 172.21.15.136:42840) 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f490f7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f491010df30 0x7f491019de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:37802/0 (socket says 172.21.15.136:37802) 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.793+0000 7f490e7fc640 1 -- 172.21.15.136:0/3583191631 learned_addr learned my addr 172.21.15.136:0/3583191631 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490effd640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4910100380 0x7f4910199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490e7fc640 1 -- 172.21.15.136:0/3583191631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f491010df30 msgr2=0x7f491019de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490e7fc640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f491010df30 0x7f491019de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490e7fc640 1 -- 172.21.15.136:0/3583191631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4910100380 msgr2=0x7f4910199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490e7fc640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4910100380 0x7f4910199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490e7fc640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4908009840 con 0x7f4910100ca0 2024-03-26T05:57:19.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490f7fe640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f491010df30 0x7f491019de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490effd640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4910100380 0x7f4910199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f490e7fc640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910100ca0 0x7f4910199a90 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f490400d770 tx=0x7f490400db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904005220 con 0x7f4910100ca0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4900009560 con 0x7f4910100ca0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f4904009cd0 con 0x7f4910100ca0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904004d00 con 0x7f4910100ca0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.794+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49101a60c0 con 0x7f4910100ca0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.795+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49100688c0 con 0x7f4910100ca0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.795+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f4904009820 con 0x7f4910100ca0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.796+0000 7f48f7fff640 1 --2- 172.21.15.136:0/3583191631 >> v2:172.21.15.136:6800/3380174081 conn(0x7f48f806d9f0 0x7f48f806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.796+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(825..825 src has 263..825) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f490408d590 con 0x7f4910100ca0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.796+0000 7f490effd640 1 --2- 172.21.15.136:0/3583191631 >> v2:172.21.15.136:6800/3380174081 conn(0x7f48f806d9f0 0x7f48f806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.796+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=826}) v3 -- 0x7f48f8079670 con 0x7f4910100ca0 2024-03-26T05:57:19.792 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.796+0000 7f490effd640 1 --2- 172.21.15.136:0/3583191631 >> v2:172.21.15.136:6800/3380174081 conn(0x7f48f806d9f0 0x7f48f806feb0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f4900000c00 tx=0x7f490002c900 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:57:19.793 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.801+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4904020060 con 0x7f4910100ca0 2024-03-26T05:57:19.970 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:19.977+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f4910113020 con 0x7f4910100ca0 2024-03-26T05:57:20.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:19 smithi136 ceph-mon[38106]: pgmap v1810: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:20.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:19 smithi136 ceph-mon[29560]: pgmap v1810: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:20.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:19 smithi164 ceph-mon[33554]: pgmap v1810: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:20.804 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:20.811+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v826) v1 ==== 217+0+0 (secure 0 0 0) 0x7f49040170d0 con 0x7f4910100ca0 2024-03-26T05:57:20.807 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:20.814+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 8 ==== osd_map(826..826 src has 263..826) v4 ==== 628+0+0 (secure 0 0 0) 0x7f49040046c0 con 0x7f4910100ca0 2024-03-26T05:57:20.807 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:20.814+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=827}) v3 -- 0x7f48f807a590 con 0x7f4910100ca0 2024-03-26T05:57:20.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:20.924+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f49101132d0 con 0x7f4910100ca0 2024-03-26T05:57:21.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:20 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:57:21.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:20 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:57:21.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:20 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:57:21.811 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.817+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v827) v1 ==== 217+0+0 (secure 0 0 0) 0x7f4904020020 con 0x7f4910100ca0 2024-03-26T05:57:21.811 INFO:tasks.workunit.client.0.smithi136.stderr:set-quota max_bytes = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 2024-03-26T05:57:21.811 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.818+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 <== mon.0 v2:172.21.15.136:3300/0 10 ==== osd_map(827..827 src has 263..827) v4 ==== 628+0+0 (secure 0 0 0) 0x7f4904004e60 con 0x7f4910100ca0 2024-03-26T05:57:21.811 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.818+0000 7f48f7fff640 1 -- 172.21.15.136:0/3583191631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=828}) v3 -- 0x7f48f807a850 con 0x7f4910100ca0 2024-03-26T05:57:21.815 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 >> v2:172.21.15.136:6800/3380174081 conn(0x7f48f806d9f0 msgr2=0x7f48f806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:21.815 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 --2- 172.21.15.136:0/3583191631 >> v2:172.21.15.136:6800/3380174081 conn(0x7f48f806d9f0 0x7f48f806feb0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f4900000c00 tx=0x7f490002c900 comp rx=0 tx=0).stop 2024-03-26T05:57:21.815 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910100ca0 msgr2=0x7f4910199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:21.815 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910100ca0 0x7f4910199a90 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f490400d770 tx=0x7f490400db70 comp rx=0 tx=0).stop 2024-03-26T05:57:21.815 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 shutdown_connections 2024-03-26T05:57:21.815 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 --2- 172.21.15.136:0/3583191631 >> v2:172.21.15.136:6800/3380174081 conn(0x7f48f806d9f0 0x7f48f806feb0 unknown :-1 s=CLOSED pgs=3565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.816 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f491010df30 0x7f491019de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.816 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910100ca0 0x7f4910199a90 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.816 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 --2- 172.21.15.136:0/3583191631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4910100380 0x7f4910199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.816 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.821+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 >> 172.21.15.136:0/3583191631 conn(0x7f4910077670 msgr2=0x7f49100ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:57:21.816 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.822+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 shutdown_connections 2024-03-26T05:57:21.816 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.822+0000 7f49153c7640 1 -- 172.21.15.136:0/3583191631 wait complete. 2024-03-26T05:57:21.827 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool set-quota 78bcab27-41f1-4625-8adb-19d91c406d36 max_objects 0 2024-03-26T05:57:21.913 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 -- 172.21.15.136:0/3542057291 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 msgr2=0x7f505c1008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:21.913 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3542057291 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 0x7f505c1008d0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f504c009890 tx=0x7f504c02bb70 comp rx=0 tx=0).stop 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 -- 172.21.15.136:0/3542057291 shutdown_connections 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3542057291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f505c100e10 0x7f505c10d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3542057291 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 0x7f505c1008d0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3542057291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f505c1064f0 0x7f505c1068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 -- 172.21.15.136:0/3542057291 >> 172.21.15.136:0/3542057291 conn(0x7f505c0775b0 msgr2=0x7f505c0779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 -- 172.21.15.136:0/3542057291 shutdown_connections 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 -- 172.21.15.136:0/3542057291 wait complete. 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 Processor -- start 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.918+0000 7f5063b3a640 1 -- start start 2024-03-26T05:57:21.914 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5063b3a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 0x7f505c10dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5063b3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f505c100e10 0x7f505c10e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5063b3a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f505c1064f0 0x7f505c10e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5063b3a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1a5790 con 0x7f505c1064f0 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5063b3a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1a5900 con 0x7f505c100e10 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5063b3a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1a5ac0 con 0x7f505c100450 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 0x7f505c10dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50620d7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f505c1064f0 0x7f505c10e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 0x7f505c10dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:59620/0 (socket says 172.21.15.136:59620) 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 -- 172.21.15.136:0/3629662691 learned_addr learned my addr 172.21.15.136:0/3629662691 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50610d5640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f505c100e10 0x7f505c10e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 -- 172.21.15.136:0/3629662691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f505c100e10 msgr2=0x7f505c10e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f505c100e10 0x7f505c10e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.915 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 -- 172.21.15.136:0/3629662691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f505c1064f0 msgr2=0x7f505c10e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f505c1064f0 0x7f505c10e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5054009900 con 0x7f505c100450 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50620d7640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f505c1064f0 0x7f505c10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f50618d6640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 0x7f505c10dcf0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f505800c890 tx=0x7f505800cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058004220 con 0x7f505c100450 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5058007e60 con 0x7f505c100450 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058005690 con 0x7f505c100450 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504c009490 con 0x7f505c100450 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.919+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505c1a60f0 con 0x7f505c100450 2024-03-26T05:57:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.920+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505c068470 con 0x7f505c100450 2024-03-26T05:57:21.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.921+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f50580079b0 con 0x7f505c100450 2024-03-26T05:57:21.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.921+0000 7f5052ffd640 1 --2- 172.21.15.136:0/3629662691 >> v2:172.21.15.136:6800/3380174081 conn(0x7f504806d9f0 0x7f504806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:21.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.921+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(827..827 src has 263..827) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f505808db60 con 0x7f505c100450 2024-03-26T05:57:21.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.921+0000 7f50610d5640 1 --2- 172.21.15.136:0/3629662691 >> v2:172.21.15.136:6800/3380174081 conn(0x7f504806d9f0 0x7f504806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:21.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.921+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=828}) v3 -- 0x7f5048079670 con 0x7f505c100450 2024-03-26T05:57:21.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.922+0000 7f50610d5640 1 --2- 172.21.15.136:0/3629662691 >> v2:172.21.15.136:6800/3380174081 conn(0x7f504806d9f0 0x7f504806feb0 secure :-1 s=READY pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7f504c009860 tx=0x7f504c032340 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:57:21.919 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:21.926+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f505801d020 con 0x7f505c100450 2024-03-26T05:57:22.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:21 smithi136 ceph-mon[38106]: pgmap v1811: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:22.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:57:22.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:21 smithi136 ceph-mon[38106]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T05:57:22.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:21 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:57:22.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:21 smithi136 ceph-mon[29560]: pgmap v1811: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:22.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:57:22.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:21 smithi136 ceph-mon[29560]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T05:57:22.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:21 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:57:22.096 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:22.103+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f505c1089e0 con 0x7f505c100450 2024-03-26T05:57:22.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:21 smithi164 ceph-mon[33554]: pgmap v1811: 188 pgs: 188 active+clean; 859 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:22.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:57:22.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:21 smithi164 ceph-mon[33554]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T05:57:22.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:21 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:57:22.820 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:22.827+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 7 ==== osd_map(828..828 src has 263..828) v4 ==== 628+0+0 (secure 0 0 0) 0x7f505805eba0 con 0x7f505c100450 2024-03-26T05:57:22.820 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:22.827+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=829}) v3 -- 0x7f504807a1e0 con 0x7f505c100450 2024-03-26T05:57:22.825 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:22.832+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v828) v1 ==== 221+0+0 (secure 0 0 0) 0x7f505805a8d0 con 0x7f505c100450 2024-03-26T05:57:22.938 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:22.945+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f505c110de0 con 0x7f505c100450 2024-03-26T05:57:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:57:23.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:22 smithi136 ceph-mon[38106]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T05:57:23.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:22 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3629662691' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:23.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:22 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:23.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:57:23.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:22 smithi136 ceph-mon[29560]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T05:57:23.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:22 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3629662691' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:23.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:22 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:23.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3583191631' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:57:23.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:22 smithi164 ceph-mon[33554]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T05:57:23.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:22 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3629662691' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:23.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:22 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:23.831 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.838+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 9 ==== osd_map(829..829 src has 263..829) v4 ==== 628+0+0 (secure 0 0 0) 0x7f5058063160 con 0x7f505c100450 2024-03-26T05:57:23.831 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.838+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=830}) v3 -- 0x7f504807a7a0 con 0x7f505c100450 2024-03-26T05:57:23.835 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.842+0000 7f5052ffd640 1 -- 172.21.15.136:0/3629662691 <== mon.2 v2:172.21.15.136:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 v829) v1 ==== 221+0+0 (secure 0 0 0) 0x7f505801d020 con 0x7f505c100450 2024-03-26T05:57:23.835 INFO:tasks.workunit.client.0.smithi136.stderr:set-quota max_objects = 0 for pool 78bcab27-41f1-4625-8adb-19d91c406d36 2024-03-26T05:57:23.839 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 >> v2:172.21.15.136:6800/3380174081 conn(0x7f504806d9f0 msgr2=0x7f504806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:23.839 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3629662691 >> v2:172.21.15.136:6800/3380174081 conn(0x7f504806d9f0 0x7f504806feb0 secure :-1 s=READY pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7f504c009860 tx=0x7f504c032340 comp rx=0 tx=0).stop 2024-03-26T05:57:23.839 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 msgr2=0x7f505c10dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:23.839 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 0x7f505c10dcf0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f505800c890 tx=0x7f505800cd50 comp rx=0 tx=0).stop 2024-03-26T05:57:23.840 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 shutdown_connections 2024-03-26T05:57:23.840 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3629662691 >> v2:172.21.15.136:6800/3380174081 conn(0x7f504806d9f0 0x7f504806feb0 unknown :-1 s=CLOSED pgs=3566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:23.840 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f505c1064f0 0x7f505c10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:23.840 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f505c100e10 0x7f505c10e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:23.840 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 --2- 172.21.15.136:0/3629662691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f505c100450 0x7f505c10dcf0 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:23.840 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 >> 172.21.15.136:0/3629662691 conn(0x7f505c0775b0 msgr2=0x7f505c1031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:57:23.840 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 shutdown_connections 2024-03-26T05:57:23.840 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:23.846+0000 7f5063b3a640 1 -- 172.21.15.136:0/3629662691 wait complete. 2024-03-26T05:57:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:+ sleep 30 2024-03-26T05:57:24.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[38106]: pgmap v1814: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[38106]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/3629662691' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[29560]: pgmap v1814: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[29560]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/3629662691' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:24.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:23 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:24.220 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:57:23 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:57:24.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:23 smithi164 ceph-mon[33554]: pgmap v1814: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T05:57:24.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:57:24.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:23 smithi164 ceph-mon[33554]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T05:57:24.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:23 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/3629662691' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:24.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:23 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:57:25.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:24 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:25.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:24 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:57:25.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:24 smithi164 ceph-mon[33554]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T05:57:25.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:24 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:25.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:24 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:57:25.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:24 smithi136 ceph-mon[38106]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T05:57:25.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:24 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:25.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:24 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:57:25.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:24 smithi136 ceph-mon[29560]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T05:57:26.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:25 smithi164 ceph-mon[33554]: pgmap v1817: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:57:26.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:25 smithi136 ceph-mon[38106]: pgmap v1817: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:57:26.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:25 smithi136 ceph-mon[29560]: pgmap v1817: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:57:27.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:57:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:57:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:57:28.220 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:27 smithi164 ceph-mon[33554]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-26T05:57:28.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:27 smithi136 ceph-mon[38106]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-26T05:57:28.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:27 smithi136 ceph-mon[29560]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.1 KiB/s wr, 1 op/s 2024-03-26T05:57:29.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:28 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:29.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:28 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:29.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:28 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:30.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:29 smithi164 ceph-mon[33554]: pgmap v1819: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-03-26T05:57:30.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:29 smithi136 ceph-mon[38106]: pgmap v1819: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-03-26T05:57:30.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:29 smithi136 ceph-mon[29560]: pgmap v1819: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-03-26T05:57:32.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:31 smithi164 ceph-mon[33554]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:57:32.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:31 smithi136 ceph-mon[38106]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:57:32.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:31 smithi136 ceph-mon[29560]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T05:57:34.221 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:57:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:57:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:33 smithi164 ceph-mon[33554]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T05:57:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:33 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:57:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:33 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:34.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:33 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:34.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:33 smithi136 ceph-mon[38106]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T05:57:34.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:33 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:57:34.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:33 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:34.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:33 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:34.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:33 smithi136 ceph-mon[29560]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T05:57:34.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:33 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:57:34.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:33 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:34.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:33 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:35.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:34 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:35.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:34 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:57:35.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:34 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:57:35.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:34 smithi164 ceph-mon[33554]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:35.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:34 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:35.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:34 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:57:35.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:34 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:57:35.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:34 smithi136 ceph-mon[38106]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:35.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:34 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:35.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:34 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:57:35.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:34 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:57:35.328 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:34 smithi136 ceph-mon[29560]: from='mgr.24485 ' entity='mgr.y' 2024-03-26T05:57:36.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:35 smithi164 ceph-mon[33554]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-03-26T05:57:36.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:35 smithi136 ceph-mon[38106]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-03-26T05:57:36.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:35 smithi136 ceph-mon[29560]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-03-26T05:57:37.577 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:57:37 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:57:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:57:38.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:37 smithi164 ceph-mon[33554]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:38.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:37 smithi136 ceph-mon[38106]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:38.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:37 smithi136 ceph-mon[29560]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:40.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:39 smithi164 ceph-mon[33554]: pgmap v1824: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:40.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:39 smithi136 ceph-mon[38106]: pgmap v1824: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:40.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:39 smithi136 ceph-mon[29560]: pgmap v1824: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:42.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:41 smithi164 ceph-mon[33554]: pgmap v1825: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:42.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:41 smithi136 ceph-mon[38106]: pgmap v1825: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:42.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:41 smithi136 ceph-mon[29560]: pgmap v1825: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:44.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:43 smithi164 ceph-mon[33554]: pgmap v1826: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:44.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:43 smithi164 ceph-mon[33554]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:44.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:57:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:57:44.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:43 smithi136 ceph-mon[38106]: pgmap v1826: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:44.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:43 smithi136 ceph-mon[38106]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:44.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:43 smithi136 ceph-mon[29560]: pgmap v1826: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:44.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:43 smithi136 ceph-mon[29560]: from='mgr.24485 172.21.15.136:0/1607388998' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:57:45.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:44 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:45.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:44 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:45.227 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:44 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:46.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:45 smithi164 ceph-mon[33554]: pgmap v1827: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:46.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:45 smithi136 ceph-mon[29560]: pgmap v1827: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:46.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:45 smithi136 ceph-mon[38106]: pgmap v1827: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:47.577 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:57:47 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:57:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:57:48.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:47 smithi164 ceph-mon[33554]: pgmap v1828: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:48.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:47 smithi136 ceph-mon[38106]: pgmap v1828: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:48.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:47 smithi136 ceph-mon[29560]: pgmap v1828: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:50.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:49 smithi164 ceph-mon[33554]: pgmap v1829: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:50.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:49 smithi136 ceph-mon[38106]: pgmap v1829: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:50.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:49 smithi136 ceph-mon[29560]: pgmap v1829: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:52.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:51 smithi164 ceph-mon[33554]: pgmap v1830: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:52.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:51 smithi136 ceph-mon[38106]: pgmap v1830: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:52.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:51 smithi136 ceph-mon[29560]: pgmap v1830: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:57:53.850 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete 78bcab27-41f1-4625-8adb-19d91c406d36 78bcab27-41f1-4625-8adb-19d91c406d36 --yes-i-really-really-mean-it 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 -- 172.21.15.136:0/4170972063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 msgr2=0x7f52a010d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/4170972063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 0x7f52a010d320 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f5298009cd0 tx=0x7f529802c0f0 comp rx=0 tx=0).stop 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 -- 172.21.15.136:0/4170972063 shutdown_connections 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/4170972063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 0x7f52a010d320 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/4170972063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52a0100450 0x7f52a01008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/4170972063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52a01064f0 0x7f52a01068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 -- 172.21.15.136:0/4170972063 >> 172.21.15.136:0/4170972063 conn(0x7f52a00775b0 msgr2=0x7f52a00779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 -- 172.21.15.136:0/4170972063 shutdown_connections 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 -- 172.21.15.136:0/4170972063 wait complete. 2024-03-26T05:57:53.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.950+0000 7f52a5bf4640 1 Processor -- start 2024-03-26T05:57:53.947 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f52a5bf4640 1 -- start start 2024-03-26T05:57:53.947 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f52a5bf4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52a0100450 0x7f52a0197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:53.947 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f52a5bf4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 0x7f52a0197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:53.947 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f52a5bf4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52a01064f0 0x7f52a019bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f52a5bf4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a019c1c0 con 0x7f52a01064f0 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f52a5bf4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a019c330 con 0x7f52a0100450 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f52a5bf4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a019c4f0 con 0x7f52a0100e10 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529effd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 0x7f52a0197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52a01064f0 0x7f52a019bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529effd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 0x7f52a0197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:52320/0 (socket says 172.21.15.136:52320) 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52a01064f0 0x7f52a019bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38200/0 (socket says 172.21.15.136:38200) 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529effd640 1 -- 172.21.15.136:0/468020195 learned_addr learned my addr 172.21.15.136:0/468020195 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529f7fe640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52a0100450 0x7f52a0197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529ffff640 1 -- 172.21.15.136:0/468020195 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 msgr2=0x7f52a0197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529ffff640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 0x7f52a0197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529ffff640 1 -- 172.21.15.136:0/468020195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52a0100450 msgr2=0x7f52a0197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529ffff640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52a0100450 0x7f52a0197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529ffff640 1 -- 172.21.15.136:0/468020195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290009490 con 0x7f52a01064f0 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.951+0000 7f529f7fe640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52a0100450 0x7f52a0197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.952+0000 7f529ffff640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52a01064f0 0x7f52a019bbc0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f529802c630 tx=0x7f5298032a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:57:53.948 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.952+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298034c70 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.952+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5298009900 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.952+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5298003e20 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.952+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52980047c0 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.952+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a01a3e70 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.953+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a0068470 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.953+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7f529803b040 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.954+0000 7f529cff9640 1 --2- 172.21.15.136:0/468020195 >> v2:172.21.15.136:6800/3380174081 conn(0x7f528806d9f0 0x7f528806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.954+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(829..829 src has 263..829) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f52980b0000 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.954+0000 7f529f7fe640 1 --2- 172.21.15.136:0/468020195 >> v2:172.21.15.136:6800/3380174081 conn(0x7f528806d9f0 0x7f528806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.954+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=830}) v3 -- 0x7f5288079670 con 0x7f52a01064f0 2024-03-26T05:57:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.954+0000 7f529f7fe640 1 --2- 172.21.15.136:0/468020195 >> v2:172.21.15.136:6800/3380174081 conn(0x7f528806d9f0 0x7f528806feb0 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f5290009d30 tx=0x7f5290005f90 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:57:53.952 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:53.959+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f529807cd70 con 0x7f52a01064f0 2024-03-26T05:57:54.128 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:54.134+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52a010fc70 con 0x7f52a01064f0 2024-03-26T05:57:54.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:53 smithi164 ceph-mon[33554]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:54.221 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:57:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug there is no tcmu-runner data available 2024-03-26T05:57:54.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:53 smithi136 ceph-mon[38106]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:54.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:53 smithi136 ceph-mon[29560]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:57:54.899 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:54.905+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]=0 pool '78bcab27-41f1-4625-8adb-19d91c406d36' removed v830) v1 ==== 248+0+0 (secure 0 0 0) 0x7f5298039100 con 0x7f52a01064f0 2024-03-26T05:57:54.899 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:54.905+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 8 ==== osd_map(830..830 src has 263..830) v4 ==== 296+0+0 (secure 0 0 0) 0x7f5298081040 con 0x7f52a01064f0 2024-03-26T05:57:54.899 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:54.905+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=831}) v3 -- 0x7f528807a2f0 con 0x7f52a01064f0 2024-03-26T05:57:55.015 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.021+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52a019c680 con 0x7f52a01064f0 2024-03-26T05:57:55.015 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.021+0000 7f529cff9640 1 -- 172.21.15.136:0/468020195 <== mon.0 v2:172.21.15.136:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]=0 pool '78bcab27-41f1-4625-8adb-19d91c406d36' does not exist v830) v1 ==== 255+0+0 (secure 0 0 0) 0x7f529804b020 con 0x7f52a01064f0 2024-03-26T05:57:55.015 INFO:tasks.workunit.client.0.smithi136.stderr:pool '78bcab27-41f1-4625-8adb-19d91c406d36' does not exist 2024-03-26T05:57:55.017 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 >> v2:172.21.15.136:6800/3380174081 conn(0x7f528806d9f0 msgr2=0x7f528806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:55.017 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/468020195 >> v2:172.21.15.136:6800/3380174081 conn(0x7f528806d9f0 0x7f528806feb0 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f5290009d30 tx=0x7f5290005f90 comp rx=0 tx=0).stop 2024-03-26T05:57:55.017 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52a01064f0 msgr2=0x7f52a019bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:55.017 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52a01064f0 0x7f52a019bbc0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f529802c630 tx=0x7f5298032a20 comp rx=0 tx=0).stop 2024-03-26T05:57:55.017 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 shutdown_connections 2024-03-26T05:57:55.017 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/468020195 >> v2:172.21.15.136:6800/3380174081 conn(0x7f528806d9f0 0x7f528806feb0 unknown :-1 s=CLOSED pgs=3567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.017 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52a01064f0 0x7f52a019bbc0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.018 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52a0100e10 0x7f52a0197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.018 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 --2- 172.21.15.136:0/468020195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52a0100450 0x7f52a0197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.018 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.023+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 >> 172.21.15.136:0/468020195 conn(0x7f52a00775b0 msgr2=0x7f52a00688c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:57:55.018 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.024+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 shutdown_connections 2024-03-26T05:57:55.018 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.024+0000 7f52a5bf4640 1 -- 172.21.15.136:0/468020195 wait complete. 2024-03-26T05:57:55.026 INFO:tasks.workunit.client.0.smithi136.stderr:+ ceph osd pool delete e5c100e4-1040-435b-858b-cfd5917b91be e5c100e4-1040-435b-858b-cfd5917b91be --yes-i-really-really-mean-it 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 -- 172.21.15.136:0/1988693079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c102800 msgr2=0x7fcb4c102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/1988693079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c102800 0x7fcb4c102be0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009900 tx=0x7fcb4802bd60 comp rx=0 tx=0).stop 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 -- 172.21.15.136:0/1988693079 shutdown_connections 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/1988693079 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb4c10bb40 0x7fcb4c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/1988693079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c103120 0x7fcb4c10b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/1988693079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c102800 0x7fcb4c102be0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 -- 172.21.15.136:0/1988693079 >> 172.21.15.136:0/1988693079 conn(0x7fcb4c0775b0 msgr2=0x7fcb4c0779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 -- 172.21.15.136:0/1988693079 shutdown_connections 2024-03-26T05:57:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.115+0000 7fcb53d3a640 1 -- 172.21.15.136:0/1988693079 wait complete. 2024-03-26T05:57:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb53d3a640 1 Processor -- start 2024-03-26T05:57:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb53d3a640 1 -- start start 2024-03-26T05:57:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb53d3a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb4c102800 0x7fcb4c1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb53d3a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c103120 0x7fcb4c197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb53d3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c10bb40 0x7fcb4c19bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb53d3a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c19c190 con 0x7fcb4c103120 2024-03-26T05:57:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb53d3a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c19c300 con 0x7fcb4c10bb40 2024-03-26T05:57:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb53d3a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c19c4c0 con 0x7fcb4c102800 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb4c102800 0x7fcb4c1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb4c102800 0x7fcb4c1972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:52340/0 (socket says 172.21.15.136:52340) 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 -- 172.21.15.136:0/796174922 learned_addr learned my addr 172.21.15.136:0/796174922 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb512d5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c103120 0x7fcb4c197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb522d7640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c10bb40 0x7fcb4c19bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 -- 172.21.15.136:0/796174922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c10bb40 msgr2=0x7fcb4c19bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c10bb40 0x7fcb4c19bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 -- 172.21.15.136:0/796174922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c103120 msgr2=0x7fcb4c197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c103120 0x7fcb4c197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 -- 172.21.15.136:0/796174922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44009840 con 0x7fcb4c102800 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb522d7640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c10bb40 0x7fcb4c19bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb512d5640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c103120 0x7fcb4c197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.116+0000 7fcb51ad6640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb4c102800 0x7fcb4c1972d0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7fcb48005940 tx=0x7fcb480040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:57:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.117+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb480044c0 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.117+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fcb48003aa0 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.117+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb48009560 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.117+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48005500 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.117+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c1a3e40 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.118+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c068470 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.118+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 22) v1 ==== 91422+0+0 (secure 0 0 0) 0x7fcb480035f0 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.118+0000 7fcb42ffd640 1 --2- 172.21.15.136:0/796174922 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcb3806d9f0 0x7fcb3806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.118+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(830..830 src has 263..830) v4 ==== 7762+0+0 (secure 0 0 0) 0x7fcb480b0280 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.118+0000 7fcb512d5640 1 --2- 172.21.15.136:0/796174922 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcb3806d9f0 0x7fcb3806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.119+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=831}) v3 -- 0x7fcb38078e50 con 0x7fcb4c102800 2024-03-26T05:57:55.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.119+0000 7fcb512d5640 1 --2- 172.21.15.136:0/796174922 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcb3806d9f0 0x7fcb3806feb0 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c077d00 tx=0x7fcb3c00a2d0 comp rx=0 tx=0).ready entity=mgr.24485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:57:55.116 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.123+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcb48077e60 con 0x7fcb4c102800 2024-03-26T05:57:55.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:54 smithi164 ceph-mon[33554]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:55.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:54 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:55.293 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.299+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcb4c1063d0 con 0x7fcb4c102800 2024-03-26T05:57:55.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:54 smithi136 ceph-mon[38106]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:55.327 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:54 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:55.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:54 smithi136 ceph-mon[29560]: from='client.24446 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:57:55.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:54 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:55.901 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.907+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 7 ==== osd_map(831..831 src has 263..831) v4 ==== 296+0+0 (secure 0 0 0) 0x7fcb48003c10 con 0x7fcb4c102800 2024-03-26T05:57:55.901 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.907+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=832}) v3 -- 0x7fcb380796a0 con 0x7fcb4c102800 2024-03-26T05:57:55.907 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:55.913+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]=0 pool 'e5c100e4-1040-435b-858b-cfd5917b91be' removed v831) v1 ==== 248+0+0 (secure 0 0 0) 0x7fcb48043020 con 0x7fcb4c102800 2024-03-26T05:57:56.018 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.025+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcb4c1063d0 con 0x7fcb4c102800 2024-03-26T05:57:56.019 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.026+0000 7fcb42ffd640 1 -- 172.21.15.136:0/796174922 <== mon.2 v2:172.21.15.136:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]=0 pool 'e5c100e4-1040-435b-858b-cfd5917b91be' does not exist v831) v1 ==== 255+0+0 (secure 0 0 0) 0x7fcb48077e60 con 0x7fcb4c102800 2024-03-26T05:57:56.019 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'e5c100e4-1040-435b-858b-cfd5917b91be' does not exist 2024-03-26T05:57:56.022 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcb3806d9f0 msgr2=0x7fcb3806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:56.022 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/796174922 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcb3806d9f0 0x7fcb3806feb0 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c077d00 tx=0x7fcb3c00a2d0 comp rx=0 tx=0).stop 2024-03-26T05:57:56.022 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb4c102800 msgr2=0x7fcb4c1972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:57:56.022 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb4c102800 0x7fcb4c1972d0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7fcb48005940 tx=0x7fcb480040c0 comp rx=0 tx=0).stop 2024-03-26T05:57:56.022 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 shutdown_connections 2024-03-26T05:57:56.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/796174922 >> v2:172.21.15.136:6800/3380174081 conn(0x7fcb3806d9f0 0x7fcb3806feb0 unknown :-1 s=CLOSED pgs=3568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:56.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c10bb40 0x7fcb4c19bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:56.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb4c103120 0x7fcb4c197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:56.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 --2- 172.21.15.136:0/796174922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb4c102800 0x7fcb4c1972d0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:57:56.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 >> 172.21.15.136:0/796174922 conn(0x7fcb4c0775b0 msgr2=0x7fcb4c1035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:57:56.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 shutdown_connections 2024-03-26T05:57:56.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-03-26T05:57:56.028+0000 7fcb53d3a640 1 -- 172.21.15.136:0/796174922 wait complete. 2024-03-26T05:57:56.031 INFO:tasks.workunit.client.0.smithi136.stdout:OK 2024-03-26T05:57:56.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo OK 2024-03-26T05:57:56.033 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T05:57:56.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T05:57:56.101 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-26T05:57:56.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[38106]: pgmap v1832: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[38106]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/796174922' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[29560]: pgmap v1832: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[29560]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T05:57:56.165 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.166 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/796174922' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.166 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:55 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:55 smithi164 ceph-mon[33554]: pgmap v1832: 188 pgs: 188 active+clean; 871 KiB data, 901 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:57:56.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:57:56.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:55 smithi164 ceph-mon[33554]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T05:57:56.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/468020195' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "78bcab27-41f1-4625-8adb-19d91c406d36", "pool2": "78bcab27-41f1-4625-8adb-19d91c406d36", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:55 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/796174922' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:55 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:56.430 DEBUG:teuthology.parallel:result is None 2024-03-26T05:57:56.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T05:57:56.497 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T05:57:56.497 DEBUG:teuthology.orchestra.run.smithi136:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T05:57:56.567 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T05:57:56.567 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T05:57:56.577 INFO:tasks.cephadm:Teardown begin 2024-03-26T05:57:56.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:57:56.634 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:57:56.669 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T05:57:56.669 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T05:57:56.689 DEBUG:teuthology.orchestra.run.smithi164:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T05:57:56.726 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T05:57:56.727 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T05:57:56.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.a 2024-03-26T05:57:57.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[38106]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[38106]: osdmap e831: 8 total, 8 up, 8 in 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[38106]: from='client.? 172.21.15.136:0/796174922' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[38106]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:56 smithi136 systemd[1]: Stopping Ceph mon.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[29560]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[29560]: osdmap e831: 8 total, 8 up, 8 in 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[29560]: from='client.? 172.21.15.136:0/796174922' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:57.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:56 smithi136 ceph-mon[29560]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:57.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:56 smithi164 ceph-mon[33554]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:57:57.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:57:57.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:56 smithi164 ceph-mon[33554]: osdmap e831: 8 total, 8 up, 8 in 2024-03-26T05:57:57.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:56 smithi164 ceph-mon[33554]: from='client.? 172.21.15.136:0/796174922' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:57.221 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:56 smithi164 ceph-mon[33554]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "e5c100e4-1040-435b-858b-cfd5917b91be", "pool2": "e5c100e4-1040-435b-858b-cfd5917b91be", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:57:57.327 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:57:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: ::ffff:172.21.15.164 - - [26/Mar/2024:05:57:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:57:57.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a[29556]: 2024-03-26T05:57:57.057+0000 7fb3c297a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:57:57.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:57 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a[29556]: 2024-03-26T05:57:57.057+0000 7fb3c297a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T05:57:57.327 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:57 smithi136 podman[152837]: 2024-03-26 05:57:57.180188096 +0000 UTC m=+0.258830923 container died 8d7627f7b5d6be93f354b15bde43f4d6eec3e4aad15622b6c78742ae60c81319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T05:57:57.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:57 smithi136 podman[152837]: 2024-03-26 05:57:57.600103847 +0000 UTC m=+0.678746673 container cleanup 8d7627f7b5d6be93f354b15bde43f4d6eec3e4aad15622b6c78742ae60c81319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T05:57:57.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Mar 26 05:57:57 smithi136 bash[152837]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-a 2024-03-26T05:57:58.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T05:57:58.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:57:58.080 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T05:57:58.080 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T05:57:58.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.c 2024-03-26T05:57:58.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:58 smithi136 systemd[1]: Stopping Ceph mon.c for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:57:58.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:58 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-c[38101]: 2024-03-26T05:57:58.421+0000 7fc44bea7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:57:58.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:58 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-c[38101]: 2024-03-26T05:57:58.421+0000 7fc44bea7700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T05:57:58.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:58 smithi136 podman[152958]: 2024-03-26 05:57:58.493198109 +0000 UTC m=+0.187339166 container died 9a92e0eeb3f7ab6b9dd342808b0509df18b4c93dd538996d81f337c9a3b53c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T05:57:59.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:58 smithi136 podman[152958]: 2024-03-26 05:57:58.855293737 +0000 UTC m=+0.549434783 container cleanup 9a92e0eeb3f7ab6b9dd342808b0509df18b4c93dd538996d81f337c9a3b53c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T05:57:59.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Mar 26 05:57:58 smithi136 bash[152958]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-c 2024-03-26T05:57:59.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T05:57:59.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:57:59.301 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T05:57:59.301 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T05:57:59.301 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.b 2024-03-26T05:57:59.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:59 smithi164 systemd[1]: Stopping Ceph mon.b for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:57:59.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-b[33550]: 2024-03-26T05:57:59.615+0000 7f2cb7e5a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:57:59.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-b[33550]: 2024-03-26T05:57:59.615+0000 7f2cb7e5a700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T05:57:59.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:57:59 smithi164 podman[75345]: 2024-03-26 05:57:59.800596797 +0000 UTC m=+0.293322946 container died 3ae84517558222e3d3b92d696d0fc75ec974b952c1b7f2ee877d5d284b14fbc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-b, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:58:00.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:58:00 smithi164 podman[75345]: 2024-03-26 05:58:00.28274452 +0000 UTC m=+0.775470668 container cleanup 3ae84517558222e3d3b92d696d0fc75ec974b952c1b7f2ee877d5d284b14fbc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:58:00.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:58:00 smithi164 bash[75345]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-b 2024-03-26T05:58:00.678 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T05:58:00.703 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:58:00 smithi164 podman[75359]: 2024-03-26 05:58:00.459257792 +0000 UTC m=+0.662067251 container remove 3ae84517558222e3d3b92d696d0fc75ec974b952c1b7f2ee877d5d284b14fbc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:58:00.703 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:58:00 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-26T05:58:00.703 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:58:00 smithi164 systemd[1]: Stopped Ceph mon.b for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:58:00.703 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 26 05:58:00 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mon.b.service: Consumed 36.238s CPU time. 2024-03-26T05:58:00.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:00.738 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T05:58:00.738 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T05:58:00.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.y 2024-03-26T05:58:01.052 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:58:00 smithi136 systemd[1]: Stopping Ceph mgr.y for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:01.327 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:58:01 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:58:01.058+0000 7f21c1cac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:01.327 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:58:01 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y[29793]: 2024-03-26T05:58:01.058+0000 7f21c1cac700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-26T05:58:01.327 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:58:01 smithi136 podman[153080]: 2024-03-26 05:58:01.091423239 +0000 UTC m=+0.138734576 container died 00c92f6e2086c6feb56e24f2403693939ac920c0e68af51d9cbb66b43f774cbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T05:58:01.621 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:58:01 smithi136 podman[153080]: 2024-03-26 05:58:01.426658823 +0000 UTC m=+0.473970155 container cleanup 00c92f6e2086c6feb56e24f2403693939ac920c0e68af51d9cbb66b43f774cbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T05:58:01.621 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Mar 26 05:58:01 smithi136 bash[153080]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-y 2024-03-26T05:58:01.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T05:58:01.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:01.879 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T05:58:01.879 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T05:58:01.879 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.x 2024-03-26T05:58:02.221 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:58:01 smithi164 systemd[1]: Stopping Ceph mgr.x for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:02.721 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:58:02 smithi164 podman[75468]: 2024-03-26 05:58:02.476695936 +0000 UTC m=+0.394652391 container died 966a6b88a8afe00c01d24bda80db9316f9cd69f692e80f50cf30c714419b52bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124) 2024-03-26T05:58:03.126 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:58:02 smithi164 podman[75468]: 2024-03-26 05:58:02.862164959 +0000 UTC m=+0.780121422 container cleanup 966a6b88a8afe00c01d24bda80db9316f9cd69f692e80f50cf30c714419b52bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:58:03.126 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Mar 26 05:58:02 smithi164 bash[75468]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-mgr-x 2024-03-26T05:58:03.271 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T05:58:03.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:03.307 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T05:58:03.307 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T05:58:03.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.0 2024-03-26T05:58:03.615 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:03 smithi136 systemd[1]: Stopping Ceph osd.0 for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:04.077 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:03 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0[42137]: 2024-03-26T05:58:03.621+0000 7f01eb3d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:04.077 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:03 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0[42137]: 2024-03-26T05:58:03.621+0000 7f01eb3d3700 -1 osd.0 831 *** Got signal Terminated *** 2024-03-26T05:58:04.077 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:03 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0[42137]: 2024-03-26T05:58:03.621+0000 7f01eb3d3700 -1 osd.0 831 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:58:09.416 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:09 smithi136 podman[153204]: 2024-03-26 05:58:09.144341492 +0000 UTC m=+5.640156220 container died 6985cb14a2fe0097430f3ad8602f36fdad65b17b6efa4f168c4fed36c0ed6611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T05:58:09.827 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:09 smithi136 podman[153204]: 2024-03-26 05:58:09.522246089 +0000 UTC m=+6.018060812 container cleanup 6985cb14a2fe0097430f3ad8602f36fdad65b17b6efa4f168c4fed36c0ed6611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T05:58:09.827 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:09 smithi136 bash[153204]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0 2024-03-26T05:58:10.268 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:09 smithi136 podman[153221]: 2024-03-26 05:58:09.949093819 +0000 UTC m=+0.799465727 container remove 6985cb14a2fe0097430f3ad8602f36fdad65b17b6efa4f168c4fed36c0ed6611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T05:58:10.529 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:10 smithi136 podman[153311]: 2024-03-26 05:58:10.274904639 +0000 UTC m=+0.104174424 container create 05139d6f7fc3dbc213700341bce91d8715c8bd521ef27f096f2da503ada8ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T05:58:10.529 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:10 smithi136 podman[153311]: 2024-03-26 05:58:10.188192118 +0000 UTC m=+0.017461920 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:58:10.819 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:10 smithi136 podman[153311]: 2024-03-26 05:58:10.53570107 +0000 UTC m=+0.364970854 container init 05139d6f7fc3dbc213700341bce91d8715c8bd521ef27f096f2da503ada8ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T05:58:10.819 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:10 smithi136 podman[153311]: 2024-03-26 05:58:10.539704251 +0000 UTC m=+0.368974036 container start 05139d6f7fc3dbc213700341bce91d8715c8bd521ef27f096f2da503ada8ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T05:58:10.819 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:10 smithi136 podman[153311]: 2024-03-26 05:58:10.644323698 +0000 UTC m=+0.473593482 container attach 05139d6f7fc3dbc213700341bce91d8715c8bd521ef27f096f2da503ada8ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True) 2024-03-26T05:58:11.077 INFO:journalctl@ceph.osd.0.smithi136.stdout:Mar 26 05:58:10 smithi136 podman[153311]: 2024-03-26 05:58:10.826242687 +0000 UTC m=+0.655512476 container died 05139d6f7fc3dbc213700341bce91d8715c8bd521ef27f096f2da503ada8ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-03-26T05:58:11.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T05:58:11.577 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:11.577 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T05:58:11.578 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T05:58:11.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.1 2024-03-26T05:58:11.827 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:11 smithi136 systemd[1]: Stopping Ceph osd.1 for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:12.327 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:11 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1[47991]: 2024-03-26T05:58:11.926+0000 7f74915e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:12.327 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:11 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1[47991]: 2024-03-26T05:58:11.926+0000 7f74915e9700 -1 osd.1 831 *** Got signal Terminated *** 2024-03-26T05:58:12.327 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:11 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1[47991]: 2024-03-26T05:58:11.926+0000 7f74915e9700 -1 osd.1 831 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:58:17.554 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:17 smithi136 podman[153426]: 2024-03-26 05:58:17.281820506 +0000 UTC m=+5.455912271 container died 1c8b2bd8975a6b3f07f8728706ad5460fcf2abceb597bf03c12e85f8dde71c33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:58:17.827 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:17 smithi136 podman[153426]: 2024-03-26 05:58:17.659818253 +0000 UTC m=+5.833910026 container cleanup 1c8b2bd8975a6b3f07f8728706ad5460fcf2abceb597bf03c12e85f8dde71c33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T05:58:17.827 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:17 smithi136 bash[153426]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1 2024-03-26T05:58:18.267 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:17 smithi136 podman[153448]: 2024-03-26 05:58:17.944528164 +0000 UTC m=+0.659916123 container remove 1c8b2bd8975a6b3f07f8728706ad5460fcf2abceb597bf03c12e85f8dde71c33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T05:58:18.541 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:18 smithi136 podman[153541]: 2024-03-26 05:58:18.177080453 +0000 UTC m=+0.018084333 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:58:18.541 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:18 smithi136 podman[153541]: 2024-03-26 05:58:18.295359771 +0000 UTC m=+0.136363678 container create f703a2f2456d2a562e5332596e3018cf34a71b68d1e45fba09e42e6023f3ac16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T05:58:18.541 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:18 smithi136 podman[153541]: 2024-03-26 05:58:18.547841995 +0000 UTC m=+0.388845858 container init f703a2f2456d2a562e5332596e3018cf34a71b68d1e45fba09e42e6023f3ac16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T05:58:18.827 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:18 smithi136 podman[153541]: 2024-03-26 05:58:18.550879367 +0000 UTC m=+0.391883228 container start f703a2f2456d2a562e5332596e3018cf34a71b68d1e45fba09e42e6023f3ac16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T05:58:18.827 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:18 smithi136 podman[153541]: 2024-03-26 05:58:18.639685648 +0000 UTC m=+0.480689531 container attach f703a2f2456d2a562e5332596e3018cf34a71b68d1e45fba09e42e6023f3ac16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T05:58:19.135 INFO:journalctl@ceph.osd.1.smithi136.stdout:Mar 26 05:58:18 smithi136 podman[153541]: 2024-03-26 05:58:18.842321534 +0000 UTC m=+0.683325399 container died f703a2f2456d2a562e5332596e3018cf34a71b68d1e45fba09e42e6023f3ac16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T05:58:19.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T05:58:19.482 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:19.482 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T05:58:19.482 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T05:58:19.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.2 2024-03-26T05:58:19.827 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:19 smithi136 systemd[1]: Stopping Ceph osd.2 for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:20.327 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:19 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2[53595]: 2024-03-26T05:58:19.837+0000 7f8ee489f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:20.327 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:19 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2[53595]: 2024-03-26T05:58:19.837+0000 7f8ee489f700 -1 osd.2 831 *** Got signal Terminated *** 2024-03-26T05:58:20.327 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:19 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2[53595]: 2024-03-26T05:58:19.837+0000 7f8ee489f700 -1 osd.2 831 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:58:25.683 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:25 smithi136 podman[153646]: 2024-03-26 05:58:25.367308385 +0000 UTC m=+5.664607863 container died b51dd1fa23195b60a64ca2eed38ce0e568b4c70f120e727aeee895d1fed08281 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T05:58:25.683 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:25 smithi136 podman[153646]: 2024-03-26 05:58:25.690068649 +0000 UTC m=+5.987368136 container cleanup b51dd1fa23195b60a64ca2eed38ce0e568b4c70f120e727aeee895d1fed08281 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T05:58:26.077 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:25 smithi136 bash[153646]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2 2024-03-26T05:58:26.077 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:25 smithi136 podman[153662]: 2024-03-26 05:58:25.899590483 +0000 UTC m=+0.529475652 container remove b51dd1fa23195b60a64ca2eed38ce0e568b4c70f120e727aeee895d1fed08281 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T05:58:26.577 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:26 smithi136 podman[153756]: 2024-03-26 05:58:26.201212196 +0000 UTC m=+0.018377384 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:58:26.577 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:26 smithi136 podman[153756]: 2024-03-26 05:58:26.308842407 +0000 UTC m=+0.126007589 container create d844bcd14cec087bcf944f335ae7ec290bfdfa345f1332c1212d61d748358008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:58:26.578 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:26 smithi136 podman[153756]: 2024-03-26 05:58:26.53621216 +0000 UTC m=+0.353377339 container init d844bcd14cec087bcf944f335ae7ec290bfdfa345f1332c1212d61d748358008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T05:58:26.578 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:26 smithi136 podman[153756]: 2024-03-26 05:58:26.539443312 +0000 UTC m=+0.356608492 container start d844bcd14cec087bcf944f335ae7ec290bfdfa345f1332c1212d61d748358008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T05:58:27.077 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:26 smithi136 podman[153756]: 2024-03-26 05:58:26.611136064 +0000 UTC m=+0.428301253 container attach d844bcd14cec087bcf944f335ae7ec290bfdfa345f1332c1212d61d748358008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:58:27.077 INFO:journalctl@ceph.osd.2.smithi136.stdout:Mar 26 05:58:26 smithi136 podman[153756]: 2024-03-26 05:58:26.83141638 +0000 UTC m=+0.648581560 container died d844bcd14cec087bcf944f335ae7ec290bfdfa345f1332c1212d61d748358008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T05:58:27.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T05:58:27.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:27.471 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T05:58:27.471 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T05:58:27.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.3 2024-03-26T05:58:27.795 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:27 smithi136 systemd[1]: Stopping Ceph osd.3 for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:28.077 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3[59564]: 2024-03-26T05:58:27.800+0000 7f64d2f6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:28.077 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3[59564]: 2024-03-26T05:58:27.800+0000 7f64d2f6b700 -1 osd.3 831 *** Got signal Terminated *** 2024-03-26T05:58:28.077 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:27 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3[59564]: 2024-03-26T05:58:27.800+0000 7f64d2f6b700 -1 osd.3 831 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:58:30.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:29 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:29.855+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:30.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:29 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:29.940+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:30.808 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:30.500+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:31.221 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:30.813+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:31.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:30 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:30.976+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:31.721 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:31 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:31.514+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:32.221 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:31 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:31.839+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:32.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:31 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:31.953+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:32.793 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:32.694+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:07.621901+0000 front 2024-03-26T05:58:07.622031+0000 (oldest deadline 2024-03-26T05:58:32.321927+0000) 2024-03-26T05:58:32.793 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:32.521+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:32.793 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:58:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:58:32.591Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-03-26T05:58:32.793 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:58:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:58:32.594Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-03-26T05:58:32.793 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:58:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:58:32.594Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-03-26T05:58:32.794 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:58:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:58:32.594Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-03-26T05:58:33.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:32.799+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:33.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:32 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:32.993+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:33.502 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:33 smithi136 podman[153863]: 2024-03-26 05:58:33.207441229 +0000 UTC m=+5.507475042 container died 08cdf2e4a7941f6804c2c247d89cf35f3f86dd1767bbcdb45982cc5f6d71276a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T05:58:33.827 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:33 smithi136 podman[153863]: 2024-03-26 05:58:33.583392563 +0000 UTC m=+5.883426367 container cleanup 08cdf2e4a7941f6804c2c247d89cf35f3f86dd1767bbcdb45982cc5f6d71276a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T05:58:33.827 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:33 smithi136 bash[153863]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3 2024-03-26T05:58:33.840 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:33.727+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:07.621901+0000 front 2024-03-26T05:58:07.622031+0000 (oldest deadline 2024-03-26T05:58:32.321927+0000) 2024-03-26T05:58:33.840 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:33.506+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:34.186 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:33 smithi136 podman[153880]: 2024-03-26 05:58:33.85103911 +0000 UTC m=+0.640167347 container remove 08cdf2e4a7941f6804c2c247d89cf35f3f86dd1767bbcdb45982cc5f6d71276a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T05:58:34.221 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:33.846+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:34.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:33 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:33.978+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:34.481 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:34 smithi136 podman[153974]: 2024-03-26 05:58:34.096342921 +0000 UTC m=+0.017839987 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:58:34.481 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:34 smithi136 podman[153974]: 2024-03-26 05:58:34.201905527 +0000 UTC m=+0.123402569 container create 3535bf3a5d9bd5f78ac0a0ca0ea04c89b5ec5502d936934215e3da302df9f4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-26T05:58:34.481 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:34 smithi136 podman[153974]: 2024-03-26 05:58:34.412490013 +0000 UTC m=+0.333987062 container init 3535bf3a5d9bd5f78ac0a0ca0ea04c89b5ec5502d936934215e3da302df9f4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T05:58:34.481 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:34 smithi136 podman[153974]: 2024-03-26 05:58:34.415692631 +0000 UTC m=+0.337189675 container start 3535bf3a5d9bd5f78ac0a0ca0ea04c89b5ec5502d936934215e3da302df9f4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T05:58:34.481 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:34 smithi136 podman[153974]: 2024-03-26 05:58:34.487791142 +0000 UTC m=+0.409288186 container attach 3535bf3a5d9bd5f78ac0a0ca0ea04c89b5ec5502d936934215e3da302df9f4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T05:58:34.827 INFO:journalctl@ceph.osd.3.smithi136.stdout:Mar 26 05:58:34 smithi136 podman[153974]: 2024-03-26 05:58:34.702132994 +0000 UTC m=+0.623630055 container died 3535bf3a5d9bd5f78ac0a0ca0ea04c89b5ec5502d936934215e3da302df9f4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=) 2024-03-26T05:58:34.852 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:34 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:34.712+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:07.621901+0000 front 2024-03-26T05:58:07.622031+0000 (oldest deadline 2024-03-26T05:58:32.321927+0000) 2024-03-26T05:58:34.852 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:34 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:34.541+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:35.221 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:34 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:34.858+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:35.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:34 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:34.960+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:35.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T05:58:35.322 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:35.322 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T05:58:35.322 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T05:58:35.323 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.4 2024-03-26T05:58:35.642 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:35 smithi164 systemd[1]: Stopping Ceph osd.4 for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:35.642 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:35 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:35.556+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:35.971 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:35 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:35.829+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:35.972 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:35 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:35.647+0000 7fd55b4ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:35.972 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:35 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:35.647+0000 7fd55b4ef700 -1 osd.4 831 *** Got signal Terminated *** 2024-03-26T05:58:35.972 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:35 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:35.647+0000 7fd55b4ef700 -1 osd.4 831 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:58:35.972 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:35 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:35.747+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:07.621901+0000 front 2024-03-26T05:58:07.622031+0000 (oldest deadline 2024-03-26T05:58:32.321927+0000) 2024-03-26T05:58:36.471 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:35 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:35.983+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:36.794 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:36 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:36.726+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:07.621901+0000 front 2024-03-26T05:58:07.622031+0000 (oldest deadline 2024-03-26T05:58:32.321927+0000) 2024-03-26T05:58:36.795 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:36 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:36.528+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:37.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:36 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:36.800+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:37.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:36 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:36.981+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:37.827 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:37 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:37.678+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:07.621901+0000 front 2024-03-26T05:58:07.622031+0000 (oldest deadline 2024-03-26T05:58:32.321927+0000) 2024-03-26T05:58:37.828 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:37 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:37.678+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.822854+0000 front 2024-03-26T05:58:12.822886+0000 (oldest deadline 2024-03-26T05:58:37.522860+0000) 2024-03-26T05:58:37.828 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:37 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:37.525+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:37.828 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:37 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:37.525+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:38.221 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:37 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:37.833+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:38.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:37 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:37.833+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:38.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:37 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:37.947+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:38.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:37 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:37.947+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:38.849 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:38 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:38.527+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:38.849 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:38 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:38.527+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:38.849 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:38 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:38.647+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:07.621901+0000 front 2024-03-26T05:58:07.622031+0000 (oldest deadline 2024-03-26T05:58:32.321927+0000) 2024-03-26T05:58:38.849 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:38 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:38.647+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.822854+0000 front 2024-03-26T05:58:12.822886+0000 (oldest deadline 2024-03-26T05:58:37.522860+0000) 2024-03-26T05:58:39.221 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:38 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:38.972+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:39.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:38 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:38.972+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:39.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:38 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:38.855+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:39.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:38 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:38.855+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:39.837 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:39 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:39.691+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:07.621901+0000 front 2024-03-26T05:58:07.622031+0000 (oldest deadline 2024-03-26T05:58:32.321927+0000) 2024-03-26T05:58:39.838 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:39 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4[38582]: 2024-03-26T05:58:39.691+0000 7fd557ae3700 -1 osd.4 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.822854+0000 front 2024-03-26T05:58:12.822886+0000 (oldest deadline 2024-03-26T05:58:37.522860+0000) 2024-03-26T05:58:39.838 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:39 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:39.507+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:39.838 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:39 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:39.507+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:40.221 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:39 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:39.843+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:40.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:39 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:39.843+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:40.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:39 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:39.978+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:40.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:39 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:39.978+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:40.853 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:40 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:40.554+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:40.853 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:40 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:40.554+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:41.222 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:41 smithi164 podman[75610]: 2024-03-26 05:58:41.000598668 +0000 UTC m=+5.467552531 container died 19a595864f37c6412f4e47801877605577c0b24767e6ac2606ec1f6893317e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T05:58:41.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:40 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:40.858+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:41.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:40 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:40.858+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:41.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:41.023+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:41.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:41.023+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:41.609 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:41.508+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:41.610 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:41.508+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:41.610 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:41 smithi164 podman[75610]: 2024-03-26 05:58:41.381393295 +0000 UTC m=+5.848347192 container cleanup 19a595864f37c6412f4e47801877605577c0b24767e6ac2606ec1f6893317e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T05:58:41.610 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:41 smithi164 bash[75610]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4 2024-03-26T05:58:41.917 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:41 smithi164 podman[75629]: 2024-03-26 05:58:41.616297637 +0000 UTC m=+0.613539470 container remove 19a595864f37c6412f4e47801877605577c0b24767e6ac2606ec1f6893317e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T05:58:41.918 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:41.861+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:41.918 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:41 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:41.861+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:42.221 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:42.065+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:42.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:42.065+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:42.222 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:41 smithi164 podman[75723]: 2024-03-26 05:58:41.827420053 +0000 UTC m=+0.018494532 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:58:42.222 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:41 smithi164 podman[75723]: 2024-03-26 05:58:41.933935248 +0000 UTC m=+0.125009776 container create 152be4f1e4e64cdd085b56102aa476ec2f062993d305e8207e5eb09b2416cb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T05:58:42.222 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:42 smithi164 podman[75723]: 2024-03-26 05:58:42.169642299 +0000 UTC m=+0.360716767 container init 152be4f1e4e64cdd085b56102aa476ec2f062993d305e8207e5eb09b2416cb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T05:58:42.222 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:42 smithi164 podman[75723]: 2024-03-26 05:58:42.172782527 +0000 UTC m=+0.363856998 container start 152be4f1e4e64cdd085b56102aa476ec2f062993d305e8207e5eb09b2416cb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T05:58:42.515 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:42 smithi164 podman[75723]: 2024-03-26 05:58:42.286519696 +0000 UTC m=+0.477594166 container attach 152be4f1e4e64cdd085b56102aa476ec2f062993d305e8207e5eb09b2416cb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T05:58:42.515 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 26 05:58:42 smithi164 podman[75723]: 2024-03-26 05:58:42.46848085 +0000 UTC m=+0.659555318 container died 152be4f1e4e64cdd085b56102aa476ec2f062993d305e8207e5eb09b2416cb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:58:42.766 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:42.520+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:42.767 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:42.521+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:43.025 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:42.829+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:43.025 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:42 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:42.829+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:43.052 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T05:58:43.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:43.097 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T05:58:43.097 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T05:58:43.097 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.5 2024-03-26T05:58:43.319 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:43 smithi164 systemd[1]: Stopping Ceph osd.5 for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:43.319 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:43.077+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:43.319 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:43.077+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:43.721 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:43.451+0000 7f29b854b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:43.722 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:43.451+0000 7f29b854b700 -1 osd.5 831 *** Got signal Terminated *** 2024-03-26T05:58:43.722 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:43.451+0000 7f29b854b700 -1 osd.5 831 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:58:43.722 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:43.542+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:43.722 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:43.542+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:44.108 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:43.850+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:44.108 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:43 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:43.850+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:44.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:44 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:44.114+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:44.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:44 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:44.114+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:44.844 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:44 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:44.528+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:44.860 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:44 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:44.528+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:45.154 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:44 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:44.850+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:45.155 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:44 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:44.850+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:45.471 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:45 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:45.160+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:45.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:45 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:45.160+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:45.879 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:45 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:45.519+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:45.879 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:45 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:45.519+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:45.879 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:45 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:45.519+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:21.388108+0000 front 2024-03-26T05:58:21.388040+0000 (oldest deadline 2024-03-26T05:58:45.488049+0000) 2024-03-26T05:58:46.202 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:45 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:45.885+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:46.202 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:45 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:45.885+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:46.471 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:46 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:46.208+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:46.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:46 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:46.208+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:46.856 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:46 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:46.528+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:46.856 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:46 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:46.528+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:46.856 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:46 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:46.528+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:21.388108+0000 front 2024-03-26T05:58:21.388040+0000 (oldest deadline 2024-03-26T05:58:45.488049+0000) 2024-03-26T05:58:47.221 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:46 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:46.862+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:47.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:46 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:46.862+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:47.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:46 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:46.862+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:47.525 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:47 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:47.237+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:47.525 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:47 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:47.237+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:47.876 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:47 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:47.531+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:08.085726+0000 front 2024-03-26T05:58:08.085707+0000 (oldest deadline 2024-03-26T05:58:30.385618+0000) 2024-03-26T05:58:47.876 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:47 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:47.531+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:12.586991+0000 front 2024-03-26T05:58:12.587049+0000 (oldest deadline 2024-03-26T05:58:37.287028+0000) 2024-03-26T05:58:47.876 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:47 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5[44090]: 2024-03-26T05:58:47.531+0000 7f29b0b37700 -1 osd.5 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:21.388108+0000 front 2024-03-26T05:58:21.388040+0000 (oldest deadline 2024-03-26T05:58:45.488049+0000) 2024-03-26T05:58:48.221 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:47 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:47.882+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:48.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:47 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:47.882+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:48.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:47 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:47.882+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:48.721 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:48 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:48.244+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:48.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:48 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:48.244+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:49.013 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:48 smithi164 podman[75828]: 2024-03-26 05:58:48.767591927 +0000 UTC m=+5.430863957 container died c19ec8267be0cbb34ad6598d0c9dbddc31e315018b9a381e9df211cac3428e62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-26T05:58:49.014 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:48 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:48.926+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:49.014 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:48 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:48.926+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:49.014 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:48 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:48.926+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:49.417 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:49 smithi164 podman[75828]: 2024-03-26 05:58:49.126962857 +0000 UTC m=+5.790234887 container cleanup c19ec8267be0cbb34ad6598d0c9dbddc31e315018b9a381e9df211cac3428e62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124) 2024-03-26T05:58:49.417 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:49 smithi164 bash[75828]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5 2024-03-26T05:58:49.417 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:49 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:49.232+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:49.417 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:49 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:49.232+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:49.418 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:49 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:49.232+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:49.418 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:49 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:49.232+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:49.722 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:49 smithi164 podman[75857]: 2024-03-26 05:58:49.423901091 +0000 UTC m=+0.652796685 container remove c19ec8267be0cbb34ad6598d0c9dbddc31e315018b9a381e9df211cac3428e62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True) 2024-03-26T05:58:50.017 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:49 smithi164 podman[75946]: 2024-03-26 05:58:49.6486569 +0000 UTC m=+0.018294958 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:58:50.017 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:49 smithi164 podman[75946]: 2024-03-26 05:58:49.779525686 +0000 UTC m=+0.149163752 container create 81b94d1479f77b8b6b5fcbcafcb2b1af794fb00a41ef59f8d0166d793048111f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T05:58:50.018 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:49 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:49.970+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:50.018 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:49 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:49.970+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:50.018 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:49 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:49.970+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:50.317 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:50 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:50.263+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:50.317 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:50 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:50.263+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:50.317 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:50 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:50.263+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:50.317 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:50 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:50.263+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:50.317 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:50 smithi164 podman[75946]: 2024-03-26 05:58:50.024020476 +0000 UTC m=+0.393658534 container init 81b94d1479f77b8b6b5fcbcafcb2b1af794fb00a41ef59f8d0166d793048111f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T05:58:50.317 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:50 smithi164 podman[75946]: 2024-03-26 05:58:50.027329582 +0000 UTC m=+0.396967648 container start 81b94d1479f77b8b6b5fcbcafcb2b1af794fb00a41ef59f8d0166d793048111f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T05:58:50.318 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:50 smithi164 podman[75946]: 2024-03-26 05:58:50.107234496 +0000 UTC m=+0.476872563 container attach 81b94d1479f77b8b6b5fcbcafcb2b1af794fb00a41ef59f8d0166d793048111f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T05:58:50.596 INFO:journalctl@ceph.osd.5.smithi164.stdout:Mar 26 05:58:50 smithi164 podman[75946]: 2024-03-26 05:58:50.323441493 +0000 UTC m=+0.693079560 container died 81b94d1479f77b8b6b5fcbcafcb2b1af794fb00a41ef59f8d0166d793048111f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T05:58:50.891 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T05:58:50.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:50.933 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T05:58:50.933 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T05:58:50.934 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.6 2024-03-26T05:58:51.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:51.015+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:51.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:51.015+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:51.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:51.015+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:51.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:51 smithi164 systemd[1]: Stopping Ceph osd.6 for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:51.722 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:51.263+0000 7f71b5d61700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:51.722 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:51.263+0000 7f71b5d61700 -1 osd.6 831 *** Got signal Terminated *** 2024-03-26T05:58:51.722 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:51.263+0000 7f71b5d61700 -1 osd.6 831 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:58:51.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:51.231+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:51.723 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:51.231+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:51.723 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:51.231+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:51.723 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:51 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:51.231+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:52.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:52.056+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:52.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:52.056+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:52.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:52.056+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:52.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:52.189+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:52.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:52.189+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:52.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:52.189+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:52.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:52 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:52.189+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:53.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:53.093+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:53.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:53.093+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:53.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:53.093+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:53.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:53.205+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:53.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:53.205+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:53.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:53.205+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:53.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:53 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:53.205+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:54.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:54.112+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:54.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:54.112+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:54.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:54.112+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:54.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:54.112+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.339380+0000 front 2024-03-26T05:58:31.339346+0000 (oldest deadline 2024-03-26T05:58:53.639289+0000) 2024-03-26T05:58:54.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:54.210+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:54.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:54.210+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:54.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:54.210+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:54.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:54.210+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:58:54.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:54 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:54.210+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:55.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:55.148+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:55.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:55.148+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:55.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:55.148+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:55.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:55.148+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.339380+0000 front 2024-03-26T05:58:31.339346+0000 (oldest deadline 2024-03-26T05:58:53.639289+0000) 2024-03-26T05:58:55.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:55.228+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:55.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:55.228+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:55.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:55.228+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:55.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:55.228+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:58:55.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:55 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:55.228+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:56.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:56.116+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:05.935116+0000 front 2024-03-26T05:58:05.934893+0000 (oldest deadline 2024-03-26T05:58:29.434908+0000) 2024-03-26T05:58:56.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:56.116+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:14.536906+0000 front 2024-03-26T05:58:14.536871+0000 (oldest deadline 2024-03-26T05:58:37.436748+0000) 2024-03-26T05:58:56.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:56.116+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:24.438153+0000 front 2024-03-26T05:58:24.438140+0000 (oldest deadline 2024-03-26T05:58:46.138088+0000) 2024-03-26T05:58:56.472 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6[49654]: 2024-03-26T05:58:56.116+0000 7f71b2355700 -1 osd.6 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.339380+0000 front 2024-03-26T05:58:31.339346+0000 (oldest deadline 2024-03-26T05:58:53.639289+0000) 2024-03-26T05:58:56.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:56.234+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:56.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:56.234+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:56.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:56.234+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:56.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:56.234+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:58:56.473 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:56 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:56.234+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:56.897 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:56 smithi164 podman[76054]: 2024-03-26 05:58:56.602283918 +0000 UTC m=+5.446749604 container died 2f564ed993402ec71961fa190bbac6b206b286609fa9ea56c14ee685d1d2d0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:58:57.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:56 smithi164 podman[76054]: 2024-03-26 05:58:56.904392323 +0000 UTC m=+5.748858009 container cleanup 2f564ed993402ec71961fa190bbac6b206b286609fa9ea56c14ee685d1d2d0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T05:58:57.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:56 smithi164 bash[76054]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6 2024-03-26T05:58:57.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:57 smithi164 podman[76072]: 2024-03-26 05:58:57.139239308 +0000 UTC m=+0.536241894 container remove 2f564ed993402ec71961fa190bbac6b206b286609fa9ea56c14ee685d1d2d0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-26T05:58:57.583 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:57 smithi164 podman[76163]: 2024-03-26 05:58:57.374562819 +0000 UTC m=+0.017979991 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:58:57.583 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:57 smithi164 podman[76163]: 2024-03-26 05:58:57.481950239 +0000 UTC m=+0.125367446 container create 7ed5e09c383fdd3b1fef65df9b91540e7fe17e08c9883151686ca0c6d7a910be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:58:57.583 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:57 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:57.278+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:57.583 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:57 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:57.278+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:57.583 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:57 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:57.278+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:57.583 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:57 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:57.278+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:58:57.583 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:57 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:57.278+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:57.972 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:57 smithi164 podman[76163]: 2024-03-26 05:58:57.692540376 +0000 UTC m=+0.335957544 container init 7ed5e09c383fdd3b1fef65df9b91540e7fe17e08c9883151686ca0c6d7a910be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T05:58:57.972 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:57 smithi164 podman[76163]: 2024-03-26 05:58:57.695680278 +0000 UTC m=+0.339097441 container start 7ed5e09c383fdd3b1fef65df9b91540e7fe17e08c9883151686ca0c6d7a910be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-26T05:58:57.972 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:57 smithi164 podman[76163]: 2024-03-26 05:58:57.767466563 +0000 UTC m=+0.410883739 container attach 7ed5e09c383fdd3b1fef65df9b91540e7fe17e08c9883151686ca0c6d7a910be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T05:58:58.272 INFO:journalctl@ceph.osd.6.smithi164.stdout:Mar 26 05:58:57 smithi164 podman[76163]: 2024-03-26 05:58:57.989416143 +0000 UTC m=+0.632833310 container died 7ed5e09c383fdd3b1fef65df9b91540e7fe17e08c9883151686ca0c6d7a910be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124) 2024-03-26T05:58:58.581 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:58.297+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:58.581 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:58.297+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:58.581 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:58.297+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:58.581 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:58.297+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:58:58.581 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:58.297+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:58:58.610 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T05:58:58.652 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:58:58.652 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T05:58:58.652 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T05:58:58.652 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.7 2024-03-26T05:58:58.951 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 systemd[1]: Stopping Ceph osd.7 for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:58:59.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:58.957+0000 7f2e8848a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:58:59.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:58.957+0000 7f2e8848a700 -1 osd.7 831 *** Got signal Terminated *** 2024-03-26T05:58:59.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:58 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:58.957+0000 7f2e8848a700 -1 osd.7 831 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:58:59.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:59.293+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:58:59.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:59.293+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:58:59.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:59.293+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:58:59.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:59.293+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:58:59.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:58:59 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:58:59.293+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:59:00.624 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:00.294+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:59:00.624 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:00.294+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:59:00.624 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:00.294+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:59:00.624 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:00.294+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:59:00.624 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:00 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:00.294+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:59:01.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:01.272+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:59:01.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:01.272+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:59:01.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:01.272+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:59:01.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:01.272+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:59:01.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:01 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:01.273+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:59:02.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:02.273+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:59:02.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:02.273+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:59:02.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:02.273+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:59:02.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:02.273+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:59:02.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:02 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:02.273+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:59:03.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:03.247+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6803 osd.0 since back 2024-03-26T05:58:03.314026+0000 front 2024-03-26T05:58:03.314015+0000 (oldest deadline 2024-03-26T05:58:29.213911+0000) 2024-03-26T05:59:03.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:03.247+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6807 osd.1 since back 2024-03-26T05:58:16.616801+0000 front 2024-03-26T05:58:16.616946+0000 (oldest deadline 2024-03-26T05:58:37.716700+0000) 2024-03-26T05:59:03.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:03.247+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6811 osd.2 since back 2024-03-26T05:58:23.017630+0000 front 2024-03-26T05:58:23.017681+0000 (oldest deadline 2024-03-26T05:58:48.917548+0000) 2024-03-26T05:59:03.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:03.247+0000 7f2e80a76700 -1 osd.7 831 heartbeat_check: no reply from 172.21.15.136:6815 osd.3 since back 2024-03-26T05:58:31.818519+0000 front 2024-03-26T05:58:31.818535+0000 (oldest deadline 2024-03-26T05:58:54.118401+0000) 2024-03-26T05:59:03.722 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:03 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7[55337]: 2024-03-26T05:59:03.247+0000 7f2e80a76700 -1 osd.7 831 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14535.0:2193 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831) 2024-03-26T05:59:04.609 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:04 smithi164 podman[76268]: 2024-03-26 05:59:04.317968504 +0000 UTC m=+5.442874134 container died 8ca9d4aa48360a08b742284915ba2f24047eaad4cde8ed0ef4d775784450f206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T05:59:04.609 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:04 smithi164 podman[76268]: 2024-03-26 05:59:04.615303445 +0000 UTC m=+5.740209074 container cleanup 8ca9d4aa48360a08b742284915ba2f24047eaad4cde8ed0ef4d775784450f206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T05:59:04.972 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:04 smithi164 bash[76268]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7 2024-03-26T05:59:04.972 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:04 smithi164 podman[76298]: 2024-03-26 05:59:04.864853874 +0000 UTC m=+0.544846896 container remove 8ca9d4aa48360a08b742284915ba2f24047eaad4cde8ed0ef4d775784450f206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T05:59:05.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:05 smithi164 podman[76389]: 2024-03-26 05:59:05.079304564 +0000 UTC m=+0.018150099 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:59:05.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:05 smithi164 podman[76389]: 2024-03-26 05:59:05.190673646 +0000 UTC m=+0.129519153 container create f6a80dae58d199b1418ec348ab2526d1df9b7470ca9e3f871b3906fb842a01c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T05:59:05.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:05 smithi164 podman[76389]: 2024-03-26 05:59:05.409801871 +0000 UTC m=+0.348647390 container init f6a80dae58d199b1418ec348ab2526d1df9b7470ca9e3f871b3906fb842a01c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T05:59:05.472 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:05 smithi164 podman[76389]: 2024-03-26 05:59:05.413407841 +0000 UTC m=+0.352253351 container start f6a80dae58d199b1418ec348ab2526d1df9b7470ca9e3f871b3906fb842a01c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:59:05.972 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:05 smithi164 podman[76389]: 2024-03-26 05:59:05.501283408 +0000 UTC m=+0.440128929 container attach f6a80dae58d199b1418ec348ab2526d1df9b7470ca9e3f871b3906fb842a01c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T05:59:05.972 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 26 05:59:05 smithi164 podman[76389]: 2024-03-26 05:59:05.698243316 +0000 UTC m=+0.637088817 container died f6a80dae58d199b1418ec348ab2526d1df9b7470ca9e3f871b3906fb842a01c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T05:59:06.297 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T05:59:06.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:59:06.341 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T05:59:06.341 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T05:59:06.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T05:59:06.828 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:06 smithi136 systemd[1]: Stopping Ceph rgw.foo.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:59:06.828 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a[64255]: 2024-03-26T05:59:06.612+0000 7f09cb22c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:59:06.828 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:06 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a[64255]: 2024-03-26T05:59:06.612+0000 7f09d9aa9ac0 -1 shutting down 2024-03-26T05:59:16.881 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:16 smithi136 bash[154079]: time="2024-03-26T05:59:16Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T05:59:16.882 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:16 smithi136 podman[154079]: 2024-03-26 05:59:16.629376252 +0000 UTC m=+10.097659695 container died 522855bae350c0fb261f7994df108b50629cd819cbd16a7e2d3becdc92da64d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T05:59:17.281 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:16 smithi136 podman[154079]: 2024-03-26 05:59:16.978117677 +0000 UTC m=+10.446401134 container cleanup 522855bae350c0fb261f7994df108b50629cd819cbd16a7e2d3becdc92da64d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T05:59:17.282 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:16 smithi136 bash[154079]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T05:59:17.539 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:17 smithi136 podman[154106]: 2024-03-26 05:59:17.287820553 +0000 UTC m=+0.653404941 container remove 522855bae350c0fb261f7994df108b50629cd819cbd16a7e2d3becdc92da64d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T05:59:17.540 INFO:journalctl@ceph.rgw.foo.a.smithi136.stdout:Mar 26 05:59:17 smithi136 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T05:59:17.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T05:59:17.582 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:59:17.582 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T05:59:17.582 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T05:59:17.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T05:59:17.884 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 systemd[1]: Stopping Ceph prometheus.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:59:18.151 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.889Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:59:18.151 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T05:59:18.151 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T05:59:18.151 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a[64946]: ts=2024-03-26T05:59:17.890Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T05:59:18.152 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:17 smithi164 podman[76494]: 2024-03-26 05:59:17.900277869 +0000 UTC m=+0.118450888 container died 0d842328995192eff452599d748502b48d2f076decb7a0e2e69a8eed68b31dd4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:59:18.472 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:18 smithi164 podman[76494]: 2024-03-26 05:59:18.256680255 +0000 UTC m=+0.474853324 container cleanup 0d842328995192eff452599d748502b48d2f076decb7a0e2e69a8eed68b31dd4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:59:18.472 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Mar 26 05:59:18 smithi164 bash[76494]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T05:59:18.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T05:59:18.713 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:59:18.713 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T05:59:18.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm rm-cluster --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T05:59:18.903 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:59:20.578 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:20 smithi136 systemd[1]: Stopping Ceph alertmanager.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:59:21.028 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:20 smithi136 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a[73936]: ts=2024-03-26T05:59:20.700Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:59:21.028 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:20 smithi136 podman[154582]: 2024-03-26 05:59:20.781682123 +0000 UTC m=+0.190811963 container died 021b2158ebf0ba82fd61d18a9a36aed170e0bf50888f86c8388020ca7f4b73c2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:59:21.294 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:21 smithi136 podman[154582]: 2024-03-26 05:59:21.125172268 +0000 UTC m=+0.534302115 container cleanup 021b2158ebf0ba82fd61d18a9a36aed170e0bf50888f86c8388020ca7f4b73c2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:59:21.295 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:21 smithi136 bash[154582]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T05:59:21.578 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:21 smithi136 podman[154596]: 2024-03-26 05:59:21.301065135 +0000 UTC m=+0.518292430 container remove 021b2158ebf0ba82fd61d18a9a36aed170e0bf50888f86c8388020ca7f4b73c2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:59:21.578 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:21 smithi136 podman[154596]: 2024-03-26 05:59:21.393506241 +0000 UTC m=+0.610733535 volume remove af34d3e06d85a7df01c6884440d08df4fa9fe90e6367e4c84f79397e6d959e58 2024-03-26T05:59:22.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:21 smithi136 systemd[1]: Stopping Ceph node-exporter.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:59:22.078 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:21 smithi136 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T05:59:22.078 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:21 smithi136 systemd[1]: Stopped Ceph alertmanager.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:59:22.079 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Mar 26 05:59:21 smithi136 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 2.352s CPU time. 2024-03-26T05:59:22.357 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:22 smithi136 podman[154709]: 2024-03-26 05:59:22.106907152 +0000 UTC m=+0.096137048 container died 938b3b4f121bd496fe5280f3808c73eba60ec68fcaf87fb7546bda963e5e654d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:59:22.732 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:22 smithi136 podman[154709]: 2024-03-26 05:59:22.454439061 +0000 UTC m=+0.443668954 container cleanup 938b3b4f121bd496fe5280f3808c73eba60ec68fcaf87fb7546bda963e5e654d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:59:22.732 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:22 smithi136 bash[154709]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T05:59:22.998 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:22 smithi136 podman[154723]: 2024-03-26 05:59:22.738774369 +0000 UTC m=+0.636276472 container remove 938b3b4f121bd496fe5280f3808c73eba60ec68fcaf87fb7546bda963e5e654d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:59:22.998 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:22 smithi136 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:59:22.998 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:22 smithi136 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T05:59:22.998 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:22 smithi136 systemd[1]: Stopped Ceph node-exporter.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:59:22.998 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Mar 26 05:59:22 smithi136 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 4.986s CPU time. 2024-03-26T05:59:23.465 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm rm-cluster --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T05:59:23.655 INFO:teuthology.orchestra.run.smithi164.stdout:Deleting cluster with fsid: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T05:59:24.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:24 smithi164 systemd[1]: Stopping Ceph iscsi.iscsi.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:59:25.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:25 smithi164 ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a[60103]: debug Shutdown received 2024-03-26T05:59:35.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 bash[76904]: time="2024-03-26T05:59:35Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T05:59:35.384 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 podman[76904]: 2024-03-26 05:59:35.151493046 +0000 UTC m=+10.114965333 container died 0b4fe70eb4bce817896a66f0715e0a29bb26681baf42c439e38fbab36f8a3222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T05:59:35.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 podman[76904]: 2024-03-26 05:59:35.567874446 +0000 UTC m=+10.531346741 container cleanup 0b4fe70eb4bce817896a66f0715e0a29bb26681baf42c439e38fbab36f8a3222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux ) 2024-03-26T05:59:35.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 bash[76904]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T05:59:36.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 podman[76920]: 2024-03-26 05:59:35.768983979 +0000 UTC m=+0.613800934 container remove 0b4fe70eb4bce817896a66f0715e0a29bb26681baf42c439e38fbab36f8a3222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:59:36.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T05:59:36.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T05:59:36.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 systemd[1]: Stopped Ceph iscsi.iscsi.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:59:36.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Mar 26 05:59:35 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 4.384s CPU time. 2024-03-26T05:59:37.722 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:37 smithi164 systemd[1]: Stopping Ceph grafana.a for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:59:38.004 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:37 smithi164 podman[77189]: 2024-03-26 05:59:37.74470727 +0000 UTC m=+0.111153464 container died 5523b342ffffe806c0b7025a818c7829d18913838787e336ee4249b91dbfadd2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.openshift.expose-services=, architecture=x86_64, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-type=git, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, vendor=Red Hat, Inc., release=239.1651231664, version=8.5) 2024-03-26T05:59:38.326 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:38 smithi164 podman[77189]: 2024-03-26 05:59:38.092616208 +0000 UTC m=+0.459062385 container cleanup 5523b342ffffe806c0b7025a818c7829d18913838787e336ee4249b91dbfadd2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.openshift.expose-services=, distribution-scope=public, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=239.1651231664, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git) 2024-03-26T05:59:38.326 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:38 smithi164 bash[77189]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a 2024-03-26T05:59:38.713 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:38 smithi164 podman[77203]: 2024-03-26 05:59:38.360493821 +0000 UTC m=+0.618465711 container remove 5523b342ffffe806c0b7025a818c7829d18913838787e336ee4249b91dbfadd2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, vendor=Red Hat, Inc., release=239.1651231664, io.openshift.expose-services=, version=8.5, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.openshift.tags=base rhel8, distribution-scope=public, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-26T05:59:38.713 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:38 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:59:38.713 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:38 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T05:59:38.713 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:38 smithi164 systemd[1]: Stopped Ceph grafana.a for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:59:38.713 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Mar 26 05:59:38 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@grafana.a.service: Consumed 18.265s CPU time. 2024-03-26T05:59:38.972 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:38 smithi164 systemd[1]: Stopping Ceph node-exporter.b for d69b39a8-eb2f-11ee-95d0-87774f69a715... 2024-03-26T05:59:39.342 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:38 smithi164 podman[77318]: 2024-03-26 05:59:38.998637026 +0000 UTC m=+0.142699856 container died 9864a66ae61cb2d9a9e4e01614b6d576fc7214e90efc1e179057dd6a2d7e736a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:59:39.342 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:39 smithi164 podman[77318]: 2024-03-26 05:59:39.348245877 +0000 UTC m=+0.492308707 container cleanup 9864a66ae61cb2d9a9e4e01614b6d576fc7214e90efc1e179057dd6a2d7e736a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:59:39.608 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:39 smithi164 bash[77318]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T05:59:39.903 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:39 smithi164 podman[77330]: 2024-03-26 05:59:39.614024099 +0000 UTC m=+0.615326936 container remove 9864a66ae61cb2d9a9e4e01614b6d576fc7214e90efc1e179057dd6a2d7e736a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:59:39.903 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:39 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:59:39.903 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:39 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T05:59:39.903 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:39 smithi164 systemd[1]: Stopped Ceph node-exporter.b for d69b39a8-eb2f-11ee-95d0-87774f69a715. 2024-03-26T05:59:39.903 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Mar 26 05:59:39 smithi164 systemd[1]: ceph-d69b39a8-eb2f-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 5.006s CPU time. 2024-03-26T05:59:40.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:59:40.471 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:59:40.500 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T05:59:40.502 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075/remote/smithi136/crash 2024-03-26T05:59:40.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/crash -- . 2024-03-26T05:59:40.558 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T05:59:40.559 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-03-26T05:59:40.560 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075/remote/smithi164/crash 2024-03-26T05:59:40.561 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/crash -- . 2024-03-26T05:59:40.602 INFO:teuthology.orchestra.run.smithi164.stderr:tar: /var/lib/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T05:59:40.602 INFO:teuthology.orchestra.run.smithi164.stderr:tar: Error is not recoverable: exiting now 2024-03-26T05:59:40.604 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T05:59:40.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T05:59:40.637 INFO:teuthology.orchestra.run.smithi136.stdout:2024-03-26T05:30:00.000148+0000 mon.a (mon.0) 2465 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-26T05:59:40.637 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T05:59:40.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[SEC\]' /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T05:59:40.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[ERR\]' /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T05:59:40.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[WRN\]' /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T05:59:40.830 INFO:teuthology.orchestra.run.smithi136.stdout:2024-03-26T05:30:00.000148+0000 mon.a (mon.0) 2465 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-26T05:59:40.830 INFO:tasks.cephadm:Compressing logs... 2024-03-26T05:59:40.830 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T05:59:40.873 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T05:59:40.897 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T05:59:40.897 INFO:teuthology.orchestra.run.smithi136.stderr:: No such file or directory 2024-03-26T05:59:40.897 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T05:59:40.897 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T05:59:40.898 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T05:59:40.898 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.a.log: /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T05:59:40.898 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T05:59:40.899 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T05:59:40.899 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T05:59:40.901 INFO:teuthology.orchestra.run.smithi164.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T05:59:40.901 INFO:teuthology.orchestra.run.smithi164.stderr:: No such file or directory 2024-03-26T05:59:40.901 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T05:59:40.901 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T05:59:40.901 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T05:59:40.901 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T05:59:40.901 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T05:59:40.905 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T05:59:40.906 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T05:59:40.906 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T05:59:40.906 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/tcmu-runner.log: 67.6% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T05:59:40.906 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log 2024-03-26T05:59:40.906 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T05:59:40.906 INFO:teuthology.orchestra.run.smithi164.stderr: 85.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T05:59:40.906 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T05:59:40.906 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T05:59:40.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mgr.y.log: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T05:59:40.911 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-volume.log: 88.1% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T05:59:40.912 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.b.log: 92.3% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T05:59:40.913 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T05:59:40.913 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.0.log: 95.3% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T05:59:40.914 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T05:59:40.920 INFO:teuthology.orchestra.run.smithi164.stderr: 91.0% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T05:59:40.927 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log 2024-03-26T05:59:40.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T05:59:40.933 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T05:59:40.943 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log: 93.4% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T05:59:40.944 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T05:59:40.952 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T05:59:40.963 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-volume.log: 94.0% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T05:59:40.993 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T05:59:41.455 INFO:teuthology.orchestra.run.smithi136.stderr: 90.7% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T05:59:42.183 INFO:teuthology.orchestra.run.smithi164.stderr: 92.6% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T05:59:42.183 INFO:teuthology.orchestra.run.smithi136.stderr: 92.7% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T05:59:44.168 INFO:teuthology.orchestra.run.smithi136.stderr: 91.9% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T05:59:48.153 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T05:59:48.797 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T05:59:48.839 INFO:teuthology.orchestra.run.smithi136.stderr: 93.9% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T05:59:48.852 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T05:59:49.007 INFO:teuthology.orchestra.run.smithi136.stderr: 93.8% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T05:59:49.291 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T05:59:49.293 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T05:59:49.293 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m8.406s 2024-03-26T05:59:49.293 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m31.635s 2024-03-26T05:59:49.294 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m1.144s 2024-03-26T05:59:49.348 INFO:teuthology.orchestra.run.smithi136.stderr: 93.9% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T05:59:50.085 INFO:teuthology.orchestra.run.smithi136.stderr: 93.9% -- replaced with /var/log/ceph/d69b39a8-eb2f-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T05:59:50.087 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-26T05:59:50.087 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m9.202s 2024-03-26T05:59:50.087 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m37.355s 2024-03-26T05:59:50.087 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m1.324s 2024-03-26T05:59:50.087 INFO:tasks.cephadm:Archiving logs... 2024-03-26T05:59:50.087 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075/remote/smithi136/log 2024-03-26T05:59:50.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T05:59:55.067 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075/remote/smithi164/log 2024-03-26T05:59:55.069 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T05:59:59.670 INFO:tasks.cephadm:Removing cluster... 2024-03-26T05:59:59.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo cephadm rm-cluster --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 --force 2024-03-26T05:59:59.863 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T06:00:00.192 DEBUG:teuthology.orchestra.run.smithi164:> sudo cephadm rm-cluster --fsid d69b39a8-eb2f-11ee-95d0-87774f69a715 --force 2024-03-26T06:00:00.389 INFO:teuthology.orchestra.run.smithi164.stdout:Deleting cluster with fsid: d69b39a8-eb2f-11ee-95d0-87774f69a715 2024-03-26T06:00:00.689 INFO:tasks.cephadm:Teardown complete 2024-03-26T06:00:00.690 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T06:00:00.698 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T06:00:00.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T06:00:00.701 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T06:00:00.778 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T06:00:00.786 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T06:00:00.786 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T06:00:00.789 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T06:00:00.801 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-26T06:00:00.802 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-26T06:00:00.805 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T06:00:00.805 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-26T06:00:00.805 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +943us[ +947us] +/- 88ms 2024-03-26T06:00:00.805 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -886us[ -882us] +/- 72ms 2024-03-26T06:00:00.805 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +129us[ +129us] +/- 74ms 2024-03-26T06:00:00.805 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:00:00.807 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T06:00:00.807 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-26T06:00:00.807 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +973us[ +975us] +/- 88ms 2024-03-26T06:00:00.807 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -855us[ -853us] +/- 72ms 2024-03-26T06:00:00.807 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +173us[ +173us] +/- 74ms 2024-03-26T06:00:00.807 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:00:00.808 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T06:00:00.816 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T06:00:00.817 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T06:00:00.831 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T06:00:00.923 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T06:00:00.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T06:00:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:00:01.028 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T06:00:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:00:01.058 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T06:00:01.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T06:00:01.076 INFO:teuthology.task.internal:Duration was 3099.312329 seconds 2024-03-26T06:00:01.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T06:00:01.086 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T06:00:01.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T06:00:01.089 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T06:00:01.129 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T06:00:01.142 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T06:00:01.361 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T06:00:01.361 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-03-26T06:00:01.361 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T06:00:01.420 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-03-26T06:00:01.420 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T06:00:01.454 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T06:00:01.454 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T06:00:01.464 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T06:00:01.521 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T06:00:01.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T06:00:01.532 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T06:00:01.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T06:00:01.990 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T06:00:01.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T06:00:02.017 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T06:00:02.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T06:00:02.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:00:02.059 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:00:02.082 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-03-26T06:00:02.110 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-03-26T06:00:02.154 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:00:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:00:02.168 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:00:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:00:02.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T06:00:02.219 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T06:00:02.220 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075/remote/smithi136 2024-03-26T06:00:02.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T06:00:02.286 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075/remote/smithi164 2024-03-26T06:00:02.286 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T06:00:02.326 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T06:00:02.326 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T06:00:02.329 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T06:00:02.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T06:00:02.386 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T06:00:02.387 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T06:00:02.401 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T06:00:02.401 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T06:00:02.404 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T06:00:02.415 INFO:teuthology.orchestra.run.smithi136.stdout: 265523 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 06:00 /home/ubuntu/cephtest 2024-03-26T06:00:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 06:00 /home/ubuntu/cephtest 2024-03-26T06:00:02.430 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T06:00:02.440 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T06:00:02.468 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T06:00:02.493 DEBUG:teuthology.nuke:shortname: smithi136 2024-03-26T06:00:02.494 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T06:00:02.516 DEBUG:teuthology.nuke:shortname: smithi164 2024-03-26T06:00:02.516 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T06:00:02.519 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:59:21.852740', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFRp4++OagDBvNa6ehQ4JmsCV/v+B5kUytsBomIUZl00QzXyrQFBndK9W5yc2eGIVqbvucpVVQqy1dNt5DuKTs0='} 2024-03-26T06:00:02.537 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622075', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:59:21.854509', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJIVMUQQX43jTvJYdvLMDQdSAdHN+8pNCO3H9rul/FAsqINti7ogSOQy5l2C+nGSgFTGSx8RlU/5Skodg5aoPFE='} 2024-03-26T06:00:02.570 INFO:teuthology.orchestra.console.smithi136:Power off 2024-03-26T06:00:02.571 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T06:00:02.596 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-03-26T06:00:02.596 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:00:02.611 INFO:teuthology.orchestra.console.smithi164:Power off 2024-03-26T06:00:02.612 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T06:00:02.619 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-26T06:00:02.633 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2024-03-26T06:00:02.633 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:00:02.659 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-26T06:00:06.620 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:00:06.660 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:00:06.732 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-26T06:00:06.771 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-26T06:00:10.774 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:00:10.783 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:00:10.887 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-26T06:00:10.894 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-26T06:00:14.892 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:00:14.901 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:00:15.004 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-03-26T06:00:15.012 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2024-03-26T06:00:15.105 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-03-26T06:00:15.112 INFO:teuthology.orchestra.console.smithi164:Power off completed 2024-03-26T06:00:15.217 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} duration: 3099.31232881546 failure_reason: '"2024-03-26T05:30:00.000148+0000 mon.a (mon.0) 2465 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-26T06:00:15.217 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T06:00:15.290 INFO:teuthology.run:FAIL